HTTP Storage Plugin - Apache Drill

<!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <meta name=viewport content="width=device-width, initial-scale=1"> <title>HTTP Storage Plugin - Apache Drill</title> <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> <link href="/css/site.css" rel="stylesheet" type="text/css"/> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"/> <link rel="icon" href="/favicon.ico" type="image/x-icon"/> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> <script language="javascript" type="text/javascript" src="/js/modernizr.custom.js"></script> <script language="javascript" type="text/javascript" src="/js/script.js"></script> <script language="javascript" type="text/javascript" src="/js/drill.js"></script> </head> <body onResize="resized();"> <div class="page-wrap"> <div class="bui"></div> <div id="menu" class="mw"> <ul> <li class='toc-categories'> <a class="expand-toc-icon" href="javascript:void(0);"></a> </li> <li class="logo"><a href="/"></a></li> <li class='expand-menu'> <a href="javascript:void(0);">Menu</a> </li> <li class="clear-float"></li> <li class="nav"> <a>Language</a> <ul> <li> <a style="font-weight: bold;" href="/docs/http-storage-plugin/" >en</a> </li> <li> <a href="/zh/docs/http-storage-plugin/" >zh</a> </li> </ul> </li> <li class="apache-link"> <a href="/apacheASF/">Apache</a> </li> <li class="poweredby"> <a href="/poweredBy">Powered By</a> </li> <li class="documentation-menu"> <a href="/docs/">Documentation</a> <ul> <li><a href="/docs/getting-started/">Getting Started</a></li> <li><a href="/docs/architecture/">Architecture</a></li> <li><a href="/docs/tutorials/">Tutorials</a></li> <li><a href="/docs/drill-on-yarn/">Drill-on-YARN</a></li> <li><a href="/docs/install-drill/">Install Drill</a></li> <li><a href="/docs/configure-drill/">Configure Drill</a></li> <li><a href="/docs/connect-a-data-source/">Connect a Data Source</a></li> <li><a href="/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> <li><a href="/docs/query-data/">Query Data</a></li> <li><a href="/docs/performance-tuning/">Performance Tuning</a></li> <li><a href="/docs/log-and-debug/">Log and Debug</a></li> <li><a href="/docs/sql-reference/">SQL Reference</a></li> <li><a href="/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> <li><a href="/docs/develop-custom-functions/">Develop Custom Functions</a></li> <li><a href="/docs/troubleshooting/">Troubleshooting</a></li> <li><a href="/docs/developer-information/">Developer Information</a></li> <li><a href="/docs/release-notes/">Release Notes</a></li> <li><a href="/docs/sample-datasets/">Sample Datasets</a></li> <li><a href="/docs/project-bylaws/">Project Bylaws</a></li> <li><a href="/docs/ecosystem/">Ecosystem</a></li> </ul> </li> <li class='nav'> <a href="/community-resources/">Community</a> <ul> <li><a href="/team/">Team</a></li> <li><a href="/mailinglists/">Mailing Lists</a></li> <li><a href="/community-resources/">Community Resources</a></li> </ul> </li> <li class='nav'><a href="/faq/">FAQ</a></li> <li class='nav'><a href="/blog/">Blog</a></li> <li class="social-menu-item"><a href="https://twitter.com/apachedrill" title="apachedrill on twitter" target="_blank"><img src="/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> <li class="social-menu-item"><a href="https://join.slack.com/t/apache-drill/shared_invite/enQtNTQ4MjM1MDA3MzQ2LTJlYmUxMTRkMmUwYmQ2NTllYmFmMjU4MDk0NjYwZjBmYjg0MDZmOTE2ZDg0ZjBlYmI3Yjc4Y2I2NTQyNGVlZTc" title="Apache Drill Slack channels" target="_blank"><img src="/images/slack-logo.svg" alt="Slack logo" align="center"></a> </li> <li class='search-bar'> <form id="drill-search-form"> <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> <button type="submit"> </button> </form> </li> <li class="d"> <a href="/download/"> Download </a> </li> </ul> </div> <link href="/css/content.css" rel="stylesheet" type="text/css"> <aside class="sidebar"> <div class="docsidebar"> <div class="docsidebarwrapper"> <ul style="display: block;"> <li class="toctree-l1"><a href="javascript: void(0);">Getting Started</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/drill-introduction/">Drill Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/why-drill/">Why Drill</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Architecture</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/architecture-introduction/">Architecture Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/drill-query-execution/">Drill Query Execution</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/core-modules/">Core Modules</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/performance/">Performance</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Tutorials</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/tutorials-introduction/">Tutorials Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/drill-in-10-minutes/">Drill in 10 Minutes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/analyzing-the-yelp-academic-dataset/">Analyzing the Yelp Academic Dataset</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Learn Drill with the MapR Sandbox</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/about-the-mapr-sandbox/">About the MapR Sandbox</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-the-apache-drill-sandbox/">Installing the Apache Drill Sandbox</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/getting-to-know-the-drill-sandbox/">Getting to Know the Drill Sandbox</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/lesson-1-learn-about-the-data-set/">Lesson 1: Learn about the Data Set</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/lesson-2-run-queries-with-ansi-sql/">Lesson 2: Run Queries with ANSI SQL</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/lesson-3-run-queries-on-complex-data-types/">Lesson 3: Run Queries on Complex Data Types</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/summary/">Summary</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/analyzing-highly-dynamic-datasets/">Analyzing Highly Dynamic Datasets</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/analyzing-social-media/">Analyzing Social Media</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/analyzing-data-using-window-functions/">Analyzing Data Using Window Functions</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/orchestrating-queries-with-airflow/">Orchestrating queries with Airflow</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Drill-on-YARN</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/drill-on-yarn-introduction/">Drill-on-YARN Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/creating-a-basic-drill-cluster/">Creating a Basic Drill Cluster</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/launch-drill-under-yarn/">Launch Drill Under YARN</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/configuration-reference/">Configuration Reference</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/drill-on-yarn-command-line-tool/">Drill-on-YARN Command-Line Tool</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/using-the-drill-on-yarn-web-ui/">Using the Drill-on-YARN Web UI</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/multiple-drill-clusters/">Multiple Drill Clusters</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/enabling-web-ui-security/">Enabling Web UI Security</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/appendix-a-release-note-issues/">Appendix A: Release Note Issues</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/appendix-b-drill-env-sh-settings/">Appendix B: drill-env.sh Settings</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/appendix-c-troubleshooting/">Appendix C: Troubleshooting</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/appendix-d-recreate-the-drill-archive/">Appendix D: Recreate the Drill Archive</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Install Drill</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/install-drill-introduction/">Install Drill Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/migrating-parquet-data/">Migrating Parquet Data</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Installing Drill in Embedded Mode</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/embedded-mode-prerequisites/">Embedded Mode Prerequisites</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/running-drill-on-docker/">Running Drill on Docker</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-drill-on-linux-and-mac-os-x/">Installing Drill on Linux and Mac OS X</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/starting-drill-on-linux-and-mac-os-x/">Starting Drill on Linux and Mac OS X</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-drill-on-windows/">Installing Drill on Windows</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/starting-drill-on-windows/">Starting Drill on Windows</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Installing Drill in Distributed Mode</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/distributed-mode-prerequisites/">Distributed Mode Prerequisites</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-drill-on-the-cluster/">Installing Drill on the Cluster</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/starting-drill-in-distributed-mode/">Starting Drill in Distributed Mode</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/starting-the-web-ui/">Starting the Web UI</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/stopping-drill/">Stopping Drill</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/identifying-multiple-drill-versions-in-a-cluster/">Identifying Multiple Drill Versions in a Cluster</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Extended</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-drill-in-distributed-mode-with-gcp-dataproc/">Installing Drill in Distributed Mode with GCP Dataproc</a></li> </ul> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Configure Drill</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/configure-drill-introduction/">Configure Drill Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/configuring-drill-memory/">Configuring Drill Memory</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Securing Drill</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/securing-drill-introduction/">Securing Drill Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/secure-communication-paths/">Secure Communication Paths</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/roles-and-privileges/">Roles and Privileges</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-user-impersonation/">Configuring User Impersonation</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-inbound-impersonation/">Configuring Inbound Impersonation</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-user-impersonation-with-hive-authorization/">Configuring User Impersonation with Hive Authorization</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-user-security/">Configuring User Security</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-plain-security/">Configuring Plain Security</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-ssl-tls-for-encryption/">Configuring SSL/TLS for Encryption</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-libpam4j-as-the-pam-authenticator/">Using libpam4j as the PAM Authenticator</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-jpam-as-the-pam-authenticator/">Using jpam as the PAM Authenticator</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-htpasswd-file-authentication/">Configuring htpasswd file authentication</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-hashicorp-vault-authentication/">Configuring HashiCorp Vault authentication</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-kerberos-security/">Configuring Kerberos Security</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-drill-to-use-spnego-for-http-authentication/">Configuring Drill to use SPNEGO for HTTP Authentication</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-web-ui-and-rest-api-security/">Configuring Web UI and REST API Security</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-custom-acls-to-secure-znodes/">Configuring Custom ACLs to Secure znodes</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Configuring a Multitenant Cluster</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-a-multitenant-cluster-introduction/">Configuring a Multitenant Cluster Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-multitenant-resources/">Configuring Multitenant Resources</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-resources-for-a-shared-drillbit/">Configuring Resources for a Shared Drillbit</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Configuration Options</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/configuration-options-introduction/">Configuration Options Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/start-up-options/">Start-Up Options</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/planning-and-execution-options/">Planning and Execution Options</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/persistent-configuration-storage/">Persistent Configuration Storage</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/global-query-list/">Global Query List</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/ports-and-bind-addresses-used-by-drill/">Ports and Bind Addresses Used by Drill</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/configuring-the-drill-shell/">Configuring the Drill Shell</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/configuring-cgroups-to-control-cpu-usage/">Configuring cgroups to Control CPU Usage</a></li> </ul> <li class="toctree-l1 current_section "><a href="javascript: void(0);">Connect a Data Source</a></li> <ul class="current_section"> <li class="toctree-l2"><a class="reference internal" href="/docs/connect-a-data-source-introduction/">Connect a Data Source Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/storage-plugin-registration/">Storage Plugin Registration</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Storage Plugin Configuration</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/plugin-configuration-basics/">Plugin Configuration Basics</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-storage-plugins/">Configuring Storage Plugins</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/storage-plugin-authentication-modes/">Storage plugin authentication modes</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/workspaces/">Workspaces</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/drill-default-input-format/">Drill Default Input Format</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/file-system-storage-plugin/">File System Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/hbase-storage-plugin/">HBase Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/hive-storage-plugin/">Hive Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/rdbms-storage-plugin/">RDBMS Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/mongodb-storage-plugin/">MongoDB Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/mapr-db-format/">MapR-DB Format</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/s3-storage-plugin/">S3 Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/oci-os-storage-plugin/">OCI OS Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/opentsdb-storage-plugin/">OpenTSDB Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/kafka-storage-plugin/">Kafka Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/azure-blob-storage-plugin/">Azure Blob Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/druid-storage-plugin/">Druid Storage Plugin</a></li> <li class="toctree-l2 current"><a class="reference internal" href="/docs/http-storage-plugin/">HTTP Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/elasticsearch-storage-plugin/">ElasticSearch Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/splunk-storage-plugin/">Splunk Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/cassandra-storage-plugin/">Cassandra Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/dropbox-storage-plugin/">Dropbox Storage Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/google-sheets-storage-plugin/">Google Sheets Storage Plugin</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">ODBC/JDBC Interfaces</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/interfaces-introduction/">Interfaces Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/using-the-jdbc-driver/">Using the JDBC Driver</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/using-jdbc-with-squirrel-on-windows/">Using JDBC with SQuirreL on Windows</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Installing the ODBC Driver</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-the-driver-on-linux/">Installing the Driver on Linux</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-the-driver-on-mac-os-x/">Installing the Driver on Mac OS X</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/installing-the-driver-on-windows/">Installing the Driver on Windows</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Configuring ODBC</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/odbc-configuration-reference/">ODBC Configuration Reference</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/logging-and-tracing/">Logging and Tracing</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-odbc-on-linux/">Configuring ODBC on Linux</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-odbc-on-mac-os-x/">Configuring ODBC on Mac OS X</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-odbc-on-windows/">Configuring ODBC on Windows</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/testing-the-odbc-connection/">Testing the ODBC Connection</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Using Drill Explorer</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/drill-explorer-introduction/">Drill Explorer Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/connecting-drill-explorer-to-data/">Connecting Drill Explorer to Data</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/browsing-data-and-defining-views/">Browsing Data and Defining Views</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Using Drill with BI Tools</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/using-drill-with-bi-tools-introduction/">Using Drill with BI Tools Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/tableau-examples/">Tableau Examples</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-microstrategy-analytics-with-apache-drill/">Using MicroStrategy Analytics with Apache Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-tibco-spotfire-desktop-with-drill/">Using Tibco Spotfire Desktop with Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-tibco-spotfire-server-with-drill/">Configuring Tibco Spotfire Server with Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-qlik-sense-with-drill/">Using Qlik Sense with Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-apache-drill-with-tableau-10-2/">Using Apache Drill with Tableau 10.2</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-apache-drill-with-tableau-9-desktop/">Using Apache Drill with Tableau 9 Desktop</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-apache-drill-with-tableau-9-server/">Using Apache Drill with Tableau 9 Server</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-information-builders-webfocus-with-apache-drill/">Using Information Builders’ WebFOCUS with Apache Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-jreport-with-drill/">Configuring JReport with Drill</a></li> </ul> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Query Data</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/query-data-introduction/">Query Data Introduction</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Querying a File System</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-a-file-system-introduction/">Querying a File System Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-avro-files/">Querying Avro Files</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-json-files/">Querying JSON Files</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-parquet-files/">Querying Parquet Files</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-plain-text-files/">Querying Plain Text Files</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-directories/">Querying Directories</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-sequence-files/">Querying Sequence Files</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/querying-hbase/">Querying HBase</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Querying Complex Data</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-complex-data-introduction/">Querying Complex Data Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/sample-data-donuts/">Sample Data: Donuts</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/selecting-flat-data/">Selecting Flat Data</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/using-sql-functions-clauses-and-joins/">Using SQL Functions, Clauses, and Joins</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/selecting-nested-data-for-a-column/">Selecting Nested Data for a Column</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/selecting-multiple-columns-within-nested-data/">Selecting Multiple Columns Within Nested Data</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/querying-hive/">Querying Hive</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/querying-the-information-schema/">Querying the INFORMATION SCHEMA</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Querying Indexes</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/querying-indexes-introduction/">Querying Indexes Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/queries-that-qualify-for-index-based-query-plans/">Queries that Qualify for Index-Based Query Plans</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/types-of-indexes/">Types of Indexes</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/index-selection/">Index Selection</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/designing-indexes-for-your-queries/">Designing Indexes for Your Queries</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/configuring-index-planning/">Configuring Index Planning</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/verifying-index-use/">Verifying Index Use</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/querying-system-tables/">Querying System Tables</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/monitoring-and-canceling-queries-in-the-drill-web-ui/">Monitoring and Canceling Queries in the Drill Web UI</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Performance Tuning</a></li> <ul style="display: none"> <li class="toctree-l2"><a href="javascript: void(0);">Drill Metastore</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/using-drill-metastore/">Using Drill Metastore</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/drill-iceberg-metastore/">Drill Iceberg Metastore</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/rdbms-metastore/">RDBMS Metastore</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/mongo-metastore/">Mongo Metastore</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/performance-tuning-introduction/">Performance Tuning Introduction</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Partition Pruning</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/partition-pruning-introduction/">Partition Pruning Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/how-to-partition-data/">How to Partition Data</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/asynchronous-parquet-reader/">Asynchronous Parquet Reader</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/optimizing-parquet-metadata-reading/">Optimizing Parquet Metadata Reading</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/parquet-filter-pushdown/">Parquet Filter Pushdown</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/hive-metadata-caching/">Hive Metadata Caching</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/choosing-a-storage-format/">Choosing a Storage Format</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Query Plans and Tuning</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/query-plans-and-tuning-introduction/">Query Plans and Tuning Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/join-planning-guidelines/">Join Planning Guidelines</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/guidelines-for-optimizing-aggregation/">Guidelines for Optimizing Aggregation</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/modifying-query-planning-options/">Modifying Query Planning Options</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/sort-based-and-hash-based-memory-constrained-operators/">Sort-Based and Hash-Based Memory-Constrained Operators</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/enabling-query-queuing/">Enabling Query Queuing</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/throttling/">Throttling</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/controlling-parallelization-to-balance-performance-with-multi-tenancy/">Controlling Parallelization to Balance Performance with Multi-Tenancy</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Identifying Performance Issues</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/query-plans/">Query Plans</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/query-profiles/">Query Profiles</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Performance Tuning Reference</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/query-profile-column-descriptions/">Query Profile Column Descriptions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/physical-operators/">Physical Operators</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/monitoring-metrics/">Monitoring Metrics</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Log and Debug</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/log-and-debug-introduction/">Log and Debug Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/error-messages/">Error Messages</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/modify-logback-xml/">Modify logback.xml</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/review-the-java-stack-trace/">Review the Java Stack Trace</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/query-audit-logging/">Query Audit Logging</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">SQL Reference</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/sql-reference-introduction/">SQL Reference Introduction</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Data Types</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/supported-data-types/">Supported Data Types</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/date-time-and-timestamp/">Date, Time, and Timestamp</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/handling-different-data-types/">Handling Different Data Types</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/lexical-structure/">Lexical Structure</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/operators/">Operators</a></li> <li class="toctree-l2"><a href="javascript: void(0);">SQL Functions</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/about-sql-function-examples/">About SQL Function Examples</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/math-and-trig/">Math and Trig</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/statistical/">Statistical</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/data-type-conversion/">Data Type Conversion</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/data-type-functions/">Data Type Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/date-time-functions-and-arithmetic/">Date/Time Functions and Arithmetic</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/string-manipulation/">String Manipulation</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/aggregate-and-aggregate-statistical/">Aggregate and Aggregate Statistical</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/functions-for-handling-nulls/">Functions for Handling Nulls</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/phonetic-functions/">Phonetic Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/string-distance-functions/">String Distance Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/cryptography-functions/">Cryptography Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/sql-dialect-compatibility-functions/">SQL dialect compatibility functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/gis-functions/">GIS functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/time-series-analysis-functions/">Time Series Analysis Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/ip-networking-functions/">IP Networking functions</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">SQL Window Functions</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/sql-window-functions-introduction/">SQL Window Functions Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/aggregate-window-functions/">Aggregate Window Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/ranking-window-functions/">Ranking Window Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/value-window-functions/">Value Window Functions</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/sql-window-functions-examples/">SQL Window Functions Examples</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Nested Data Functions</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/nested-data-limitations/">Nested Data Limitations</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/flatten/">FLATTEN</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/kvgen/">KVGEN</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/repeated-count/">REPEATED_COUNT</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/repeated-contains/">REPEATED_CONTAINS</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/list-creation-functions/">COLLECT_LIST</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/query-directory-functions/">Query Directory Functions</a></li> <li class="toctree-l2"><a href="javascript: void(0);">SQL Commands</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/supported-sql-commands/">Supported SQL Commands</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/analyze-table-refresh-metadata/">ANALYZE TABLE REFRESH METADATA</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/analyze-table-compute-statistics/">ANALYZE TABLE COMPUTE STATISTICS</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/set/">SET</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/refresh-table-metadata/">REFRESH TABLE METADATA</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/reset/">RESET</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/alter-system/">ALTER SYSTEM</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/create-or-replace-schema/">CREATE OR REPLACE SCHEMA</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/create-table-as-ctas/">CREATE TABLE AS (CTAS)</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/create-temporary-table-as-cttas/">CREATE TEMPORARY TABLE AS (CTTAS)</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/create-function-using-jar/">CREATE FUNCTION USING JAR</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/partition-by-clause/">PARTITION BY Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/create-view/">CREATE VIEW</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/describe/">DESCRIBE</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/drop-function-using-jar/">DROP FUNCTION USING JAR</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/drop-table/">DROP TABLE</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/drop-view/">DROP VIEW</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/explain/">EXPLAIN</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/lateral-join/">LATERAL Join</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/select/">SELECT</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/select-list/">SELECT List</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/from-clause/">FROM Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/group-by-clause/">GROUP BY Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/having-clause/">HAVING Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/limit-clause/">LIMIT Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/offset-clause/">OFFSET Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/order-by-clause/">ORDER BY Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/set-operators/">Set Operators</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/where-clause/">WHERE Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/with-clause/">WITH Clause</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/show-databases-and-show-schemas/">SHOW DATABASES and SHOW SCHEMAS</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/show-files/">SHOW FILES</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/show-tables/">SHOW TABLES</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/use/">USE</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/pivot-operators/">Pivot Operators</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">SQL Conditional Expressions</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/case/">CASE</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/reserved-keywords/">Reserved Keywords</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/sql-extensions/">SQL Extensions</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Data Sources and File Formats</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/data-sources-and-file-formats-introduction/">Data Sources and File Formats Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/hive-to-drill-data-type-mapping/">Hive-to-Drill Data Type Mapping</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/deploying-and-using-a-hive-udf/">Deploying and Using a Hive UDF</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/parquet-format/">Parquet Format</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/logfile-plugin/">Logfile Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/json-data-model/">JSON Data Model</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/text-files-csv-tsv-psv/">Text Files: CSV, TSV, PSV</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/sequence-files/">Sequence Files</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/httpd-format-plugin/">HTTPD Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/image-metadata-format-plugin/">Image Metadata Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/syslog-format-plugin/">Syslog Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/ltsv-format-plugin/">LTSV Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/spss-format-plugin/">SPSS Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/esri-shapefile-format-plugin/">ESRI Shapefile Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/excel-format-plugin/">Excel Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/hdf5-format-plugin/">HDF5 Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/ms-access-format-plugin/">Microsoft Access Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/xml-format-plugin/">XML Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/iceberg-format-plugin/">Iceberg Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/delta-lake-format-plugin/">Delta Lake Format Plugin</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/sas-format-plugin/">SAS Format Plugin</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Develop Custom Functions</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/develop-custom-functions-introduction/">Develop Custom Functions Introduction</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/developing-a-simple-function/">Developing a Simple Function</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/tutorial-develop-a-simple-function/">Tutorial: Develop a Simple Function</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/developing-an-aggregate-function/">Developing an Aggregate Function</a></li> <li class="toctree-l2"><a href="javascript: void(0);">Adding Custom Functions to Drill</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/adding-custom-functions-to-drill-introduction/">Adding Custom Functions to Drill Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/manually-adding-custom-functions-to-drill/">Manually Adding Custom Functions to Drill</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/creating-custom-authenticators/">Creating Custom Authenticators</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/dynamic-udfs/">Dynamic UDFs</a></li> </ul> <li class="toctree-l2"><a class="reference internal" href="/docs/using-custom-functions-in-queries/">Using Custom Functions in Queries</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/custom-function-interfaces/">Custom Function Interfaces</a></li> </ul> <li class="toctree-l1"><a class="reference internal" href="/docs/troubleshooting/">Troubleshooting</a></li> <li class="toctree-l1"><a href="javascript: void(0);">Developer Information</a></li> <ul style="display: none"> <li class="toctree-l2"><a href="javascript: void(0);">REST API</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/rest-api-introduction/">REST API Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled/">Submitting Queries from the REST API when Impersonation is Enabled and Authentication is Disabled</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/use-postman-to-run-sql-queries-on-drill-data-sources/">Use Postman to Run SQL Queries on Drill Data Sources</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Develop Drill</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/compiling-drill-from-source/">Compiling Drill from Source</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/useful-information-for-drill-developers/">Useful Information for Drill Developers</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Contribute to Drill</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/apache-drill-contribution-guidelines/">Apache Drill Contribution Guidelines</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/apache-drill-contribution-ideas/">Apache Drill Contribution Ideas</a></li> </ul> <li class="toctree-l2"><a href="javascript: void(0);">Design Docs</a></li> <ul style="display: none"> <li class="toctree-l3"><a class="reference internal" href="/docs/drill-plan-syntax/">Drill Plan Syntax</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/rpc-overview/">RPC Overview</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/query-stages/">Query Stages</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/useful-research/">Useful Research</a></li> <li class="toctree-l3"><a class="reference internal" href="/docs/value-vectors/">Value Vectors</a></li> </ul> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Release Notes</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-21-2-release-notes/">Apache Drill 1.21.2 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-21-1-release-notes/">Apache Drill 1.21.1 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-21-0-release-notes/">Apache Drill 1.21.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-20-3-release-notes/">Apache Drill 1.20.3 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-20-2-release-notes/">Apache Drill 1.20.2 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-20-1-release-notes/">Apache Drill 1.20.1 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-20-0-release-notes/">Apache Drill 1.20.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-19-0-release-notes/">Apache Drill 1.19.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-18-0-release-notes/">Apache Drill 1.18.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-17-0-release-notes/">Apache Drill 1.17.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-16-0-release-notes/">Apache Drill 1.16.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-15-0-release-notes/">Apache Drill 1.15.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-14-0-release-notes/">Apache Drill 1.14.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-13-0-release-notes/">Apache Drill 1.13.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-12-0-release-notes/">Apache Drill 1.12.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-11-0-release-notes/">Apache Drill 1.11.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-10-0-release-notes/">Apache Drill 1.10.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-9-0-release-notes/">Apache Drill 1.9.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-8-0-release-notes/">Apache Drill 1.8.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-7-0-release-notes/">Apache Drill 1.7.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-6-0-release-notes/">Apache Drill 1.6.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-5-0-release-notes/">Apache Drill 1.5.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-4-0-release-notes/">Apache Drill 1.4.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-3-0-release-notes/">Apache Drill 1.3.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-2-0-release-notes/">Apache Drill 1.2.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-1-0-release-notes/">Apache Drill 1.1.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-1-0-0-release-notes/">Apache Drill 1.0.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-9-0-release-notes/">Apache Drill 0.9.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-8-0-release-notes/">Apache Drill 0.8.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-7-0-release-notes/">Apache Drill 0.7.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-6-0-release-notes/">Apache Drill 0.6.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-5-0-release-notes/">Apache Drill 0.5.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-0-4-0-release-notes/">Apache Drill 0.4.0 Release Notes</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/apache-drill-m1-release-notes-apache-drill-alpha/">Apache Drill M1 Release Notes (Apache Drill Alpha)</a></li> </ul> <li class="toctree-l1"><a href="javascript: void(0);">Sample Datasets</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/enron-emails/">Enron Emails</a></li> <li class="toctree-l2"><a class="reference internal" href="/docs/wikipedia-edit-history/">Wikipedia Edit History</a></li> </ul> <li class="toctree-l1"><a class="reference internal" href="/docs/project-bylaws/">Project Bylaws</a></li> <li class="toctree-l1"><a href="javascript: void(0);">Ecosystem</a></li> <ul style="display: none"> <li class="toctree-l2"><a class="reference internal" href="/docs/using-saiku-analytics-with-apache-drill/">Using Saiku Analytics with Apache Drill</a></li> </ul> </ul> </div> </div> </aside> <nav class="breadcrumbs"> <li><a href="/docs/">Docs</a></li> <li><a href="/docs/connect-a-data-source/">Connect a Data Source</a></li> <li>HTTP Storage Plugin</li> </nav> <div class="main-content-wrapper"> <div class="main-content"> <a class="edit-link" href="https://github.com/apache/drill-site/blob/master/_docs/en/connect-a-data-source/plugins/125-http-storage-plugin.md" target="_blank"></a> <div class="int_title left"> <h1>HTTP Storage Plugin</h1> </div>  <div class="int_text" align="left"> Introduced in release: 1.18 The HTTP storage plugin lets you query APIs over HTTP/REST. The plugin expects JSON or CSV responses. The HTTP plugin is new in Drill 1.18 and is an Alpha feature. It works well, and we enourage you to use it and provide feedback. However, we reserve the right to change the plugin based on that feedback. <h2 id="configuration">Configuration</h2> To configure the plugin, create a new storage plugin and add the following configuration options which apply to all of the connections defined in this plugin: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "type": "http", "cacheResults": true, "connections": {}, "timeout": 0, "proxyHost": null, "proxyPort": 0, "proxyType": null, "proxyUsername": null, "proxyPassword": null, "enabled": true } </code></pre></div></div> <h3 id="configuration-options">Configuration Options</h3> <table> <thead> <tr> <th>Option</th> <th>Default</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>type</td> <td>(none)</td> <td>Set to “http” to use this plugin</td> </tr> <tr> <td>cacheResults</td> <td>false</td> <td>Enables local caching of received HTTP response data</td> </tr> <tr> <td>timeout</td> <td>0</td> <td>HTTP request timeout in seconds, 0 = no timeout</td> </tr> <tr> <td>connections</td> <td>(none)</td> <td>This field contains the details for individual connections. See the section Configuring API Connections for Details</td> </tr> </tbody> </table> You can configure the HTTP storage plugin to work through an HTTP proxy. Details are listed below. <h3 id="configuring-the-api-connections">Configuring the API Connections</h3> The HTTP Storage plugin allows you to configure multiple APIS which you can query directly from this plugin. To do so, first add a <code class="language-plaintext highlighter-rouge">connections</code> parameter to the configuration. Next give the connection a name, which will be used in queries. For instance <code class="language-plaintext highlighter-rouge">stockAPI</code> or <code class="language-plaintext highlighter-rouge">jira</code>. The <code class="language-plaintext highlighter-rouge">connection</code> property can accept the following options. <h4 id="url">URL</h4> <code class="language-plaintext highlighter-rouge">url</code>: The base URL which Drill will query. <code class="language-plaintext highlighter-rouge">requireTail</code>: Set to <code class="language-plaintext highlighter-rouge">true</code> if the query must contain an additional part of the service URL as a table name, <code class="language-plaintext highlighter-rouge">false</code> if the URL needs no additional suffix other than that provided by <code class="language-plaintext highlighter-rouge">WHERE</code> clause filters. (See below.) If your service requires parameters, you have three choices. Suppose your connection is called <code class="language-plaintext highlighter-rouge">sunrise</code>. First, can include them directly in your URL if the parameters a fixed for a given service: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "url": "https://api.sunrise-sunset.org/json?lat=36.7201600&lng=-4.4203400&date=2019-10-02", "requireTail": false } </code></pre></div></div> Query your table like this: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT * FROM api.sunrise; </code></pre></div></div> Second, you can specify the base URL here and the full URL in your query. Use this form if the parameters define a table-like concept (the set of data to return): <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "url": "https://api.sunrise-sunset.org/json", "requireTail": true } </code></pre></div></div> SQL query: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT * FROM api.sunrise.`?lat=36.7201600&lng=-4.4203400&date=2019-10-02` </code></pre></div></div> If the URL requires a tail, specify it as if it were a table name. (See example below.) Drill directly appends the “tail” to the base URL to create the final URL. Third, you can use the <code class="language-plaintext highlighter-rouge">params</code> field below to specify the parameters as filters if the parameters specify which data sets to return: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "url": "https://api.sunrise-sunset.org/json", "requireTail": false, "params": ["lat", "lng", "date"] } </code></pre></div></div> SQL query: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT * FROM api.sunrise WHERE `lat` = 36.7201600 AND `lng` = -4.4203400 AND `date` = '2019-10-02' </code></pre></div></div> In this case, Drill appends the parameters to the URL, adding a question mark to separate the two. <h4 id="method">Method</h4> <code class="language-plaintext highlighter-rouge">method</code>: The request method. Must be <code class="language-plaintext highlighter-rouge">GET</code> or <code class="language-plaintext highlighter-rouge">POST</code>. Other methods are not allowed and will default to <code class="language-plaintext highlighter-rouge">GET</code>. <code class="language-plaintext highlighter-rouge">postBody</code>: Contains data, in the form of key value pairs, which are sent during a <code class="language-plaintext highlighter-rouge">POST</code> request. The post body should be in the of a block of text with key/value pairs: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "postBody": "key1=value1 key2=value2" } </code></pre></div></div> <h4 id="headers">Headers</h4> <code class="language-plaintext highlighter-rouge">headers</code>: Often APIs will require custom headers as part of the authentication. This field allows you to define key/value pairs which are submitted with the http request. The format is: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"headers": { "key1": "Value1", "key2": "Value2" } </code></pre></div></div> <h4 id="query-parmeters-as-filters">Query Parmeters as Filters</h4> <ul> <li><code class="language-plaintext highlighter-rouge">params</code>: Allows you to map SQL <code class="language-plaintext highlighter-rouge">WHERE</code> clause conditions to query parameters.</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "url": "https://api.sunrise-sunset.org/json", "params": ["lat", "lng", "date"] } </code></pre></div></div> SQL query: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT * FROM api.sunrise WHERE `lat` = 36.7201600 AND `lng` = -4.4203400 AND `date` = '2019-10-02' </code></pre></div></div> HTTP parameters are untyped; Drill converts any value you provide into a string. Drill allows you to use any data type which can convert unambiguously to a string: <code class="language-plaintext highlighter-rouge">BIT</code>, <code class="language-plaintext highlighter-rouge">INT</code>, <code class="language-plaintext highlighter-rouge">BIGINT</code>, <code class="language-plaintext highlighter-rouge">FLOAT4</code>, <code class="language-plaintext highlighter-rouge">FLOAT8</code>, <code class="language-plaintext highlighter-rouge">VARDECIMAL</code>, <code class="language-plaintext highlighter-rouge">VARCHAR</code>. The <code class="language-plaintext highlighter-rouge">BIT</code> type is translated to <code class="language-plaintext highlighter-rouge">true</code> and <code class="language-plaintext highlighter-rouge">false</code>. Note that none of the date or interval types are allowed: each of those requires formatting. Note the need for back-tick quotes around the names; <code class="language-plaintext highlighter-rouge">date</code> is a reserved word. Notice also that the date is a string because of the formatting limitation mentioned above. Only equality conditions can be translated to parameters. The above filters are translated to: <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lat=36.7201600&lng=-4.4203400&date=2019-10-02 </code></pre></div></div> If your query contains other conditions (<code class="language-plaintext highlighter-rouge">!=</code>, <code class="language-plaintext highlighter-rouge"><</code>, etc.) then those conditions are applied in Drill after the REST service returns the data. Only fields listed in the <code class="language-plaintext highlighter-rouge">params</code> config filed will become parameters, all other experssions are handled within Drill as explained above. At present, Drill requires the values to be literals (constants). Drill does not currently allow expressions. That is, the following will not become an HTTP parameter: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WHERE `lat` = 36 + 0.7201600 </code></pre></div></div> Drill will add parameters to the URL in the order listed in the config. Use this feature if the API is strict about parameter ordering. At present Drill does not enforce that parameters are provided in the query: Drill assumes parameters are optional. <h4 id="api-connection-credentials-provider">API connection credentials provider</h4> Starting from Drill 1.20.0, it is possible to use <a href="https://github.com/apache/drill/blob/master/docs/dev/PluginCredentialsProvider.md#plugin-credentials-provider">Credentials Provider</a> for specifying API connection credentials. <code class="language-plaintext highlighter-rouge">credentialsProvider</code> property should be used to configure credentials provider, similar to the way how it is used for storage plugins: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "credentialsProvider": { "credentialsProviderType": "PlainCredentialsProvider", "credentials": { "username": "user1", "password": "user1Pass" } } } </code></pre></div></div> It is possible to use any available credential provider type for API connections, including custom implementations. Please refer to <a href="https://github.com/apache/drill/blob/master/docs/dev/PluginCredentialsProvider.md#developer-notes">Developer notes</a> for more details for creating custom credential provider implementations without changing Drill’s code. <h3 id="data-path">Data Path</h3> REST responses often have structure beyond the data you want to query. For example: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "results": { "sunrise":"7:27:02 AM", "sunset":"5:05:55 PM", "solar_noon":"12:16:28 PM", "day_length":"9:38:53", "civil_twilight_begin":"6:58:14 AM", "civil_twilight_end":"5:34:43 PM", "nautical_twilight_begin":"6:25:47 AM", "nautical_twilight_end":"6:07:10 PM", "astronomical_twilight_begin":"5:54:14 AM", "astronomical_twilight_end":"6:38:43 PM" }, "status":"OK" } </code></pre></div></div> Drill can handle JSON structures such as the above; you can use SQL to obtain the results you want. However, the SQL will be simpler if we skip over the portions we don’t want and simply read the <code class="language-plaintext highlighter-rouge">results</code> fields as our SQL fields. We do that with the <code class="language-plaintext highlighter-rouge">dataPath</code> configuration: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"dataPath": "results" </code></pre></div></div> The <code class="language-plaintext highlighter-rouge">dataPath</code> can contain any number of fields, for example: <code class="language-plaintext highlighter-rouge">"response/content/data"</code>. Drill will ignore all JSON content outside of the data path. At present, there is no provision to check the <code class="language-plaintext highlighter-rouge">status</code> code in a response such as that shown above. Drill assumes that the server will uses HTTP status codes to indicate a bad request or other error. <h3 id="input-type">Input Type</h3> The REST plugin accepts three different types of input: <code class="language-plaintext highlighter-rouge">json</code>, <code class="language-plaintext highlighter-rouge">csv</code> and <code class="language-plaintext highlighter-rouge">xml</code>. The default is <code class="language-plaintext highlighter-rouge">json</code>. If you are using <code class="language-plaintext highlighter-rouge">XML</code> as a data type, there is an additional configuration option called <code class="language-plaintext highlighter-rouge">xmlDataLevel</code> which reduces the level of unneeded nesting found in XML files. You can find more information in the documentation for Drill’s XML format plugin. <h4 id="authorization">Authorization</h4> <code class="language-plaintext highlighter-rouge">authType</code>: If your API requires authentication, specify the authentication type. At the time of implementation, the plugin only supports basic authentication, however, the plugin will likely support OAUTH2 in the future. Defaults to <code class="language-plaintext highlighter-rouge">none</code>. If the <code class="language-plaintext highlighter-rouge">authType</code> is set to <code class="language-plaintext highlighter-rouge">basic</code>, <code class="language-plaintext highlighter-rouge">username</code> and <code class="language-plaintext highlighter-rouge">password</code> must be set in the configuration as well. <code class="language-plaintext highlighter-rouge">username</code>: The username for basic authentication. <code class="language-plaintext highlighter-rouge">password</code>: The password for basic authentication. <h2 id="usage">Usage</h2> This plugin is different from other plugins in that it the table component of the <code class="language-plaintext highlighter-rouge">FROM</code> clause is different. In normal Drill queries, the <code class="language-plaintext highlighter-rouge">FROM</code> clause is constructed as follows: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM <storage plugin>.<schema>.<table> </code></pre></div></div> For example, you might have: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM dfs.test.`somefile.csv` -- or FROM mongo.stats.sales_data </code></pre></div></div> The HTTP/REST plugin the <code class="language-plaintext highlighter-rouge">FROM</code> clause enables you to pass arguments to your REST call if you set the <code class="language-plaintext highlighter-rouge">requireTail</code> property to <code class="language-plaintext highlighter-rouge">true</code>. The structure is: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM <plugin>.<connection>.<arguments> --Actual example: FROM http.sunrise.`/json?lat=36.7201600&lng=-4.4203400&date=today` </code></pre></div></div> Paging through introduced in release: 1.20. Or, as explained above, you can have the URL act like a table and pass parameters using a <code class="language-plaintext highlighter-rouge">WHERE</code> clause “filter” conPaging through pitions. # Introduced in release: 1.20. <h2 id="proxy-setup">Proxy Setup</h2> ## Some users access HTTP services from behind a proxy firewall. Drill provides three ways specify proxy cnfiguration. <h3 id="proxy-environment-variables">Proxy Environment Variables</h3> Drill recognizes the usual Linux proxy environment variables: *# <code class="language-plaintext highlighter-rouge">http_proxy</code>, <code class="language-plaintext highlighter-rouge">HTTP_PROXY</code> <ul> <li><code class="language-plaintext highlighter-rouge">https_proxy</code>, <code class="language-plaintext highlighter-rouge">HTTP_PROXY</code></li> <li><code class="language-plaintext highlighter-rouge">Paging through pll_proxy</code>, <code class="language-plaintext highlighter-rouge">ALL_PROXY</code> # Introduced in release: 1.20.</li> </ul> #This technique works well if your system is already configured to handle proxies. # <h3 id="boot-configuration">Boot Configuration</h3> You can also specify proxy configuration in the <code class="language-plaintext highlighter-rouge">drill-override.conf</code> file. #See <code class="language-plaintext highlighter-rouge">drill-override-example.conf</code> for a template. Use the boot configuration is an attribute of your network environment. Doing so will ensure every Drillbit and every HTTP/HTTPS request uses the same proxy configuration. First, you can use the same form of URL you would use with the environment variables: <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> d#rill.exec.net_proxy.http_url: "http://foo.com/1234" </code></pre></div></div> There is one setting for HTTP, another for HTTPS. Alternatively, you can specify each field separately: <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drill.exec.net_proxy.http: { type: "none", # none, http, socks. Blank same as none. host: "", port: 80, user_name: "", password: "" }, </code></pre></div></div> The valid proxy types are <code class="language-plaintext highlighter-rouge">none</code>, <code class="language-plaintext highlighter-rouge">http</code> and <code class="language-plaintext highlighter-rouge">socks</code>. Blank is the same as <code class="language-plaintext highlighter-rouge">none</code>. Again, there is a parallel section for HTTPS. <h3 id="in-the-http-storage-plugin-config">In the HTTP Storage Plugin Config</h3> The final way to configure proxy is in the HTTP storage plugin itself. The proxy applies to all connections defined in that plugin. Use this approach if the proxy applies only to some external services, or if each service has a different proxy (defined by creating a separate plugin config for each service.) <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "proxy_type": "direct", "proxy_host": "", "proxy_port": 80, "proxy_user_name": "", "proxy_password": "" } </code></pre></div></div> The valid proxy types are <code class="language-plaintext highlighter-rouge">direct</code>, <code class="language-plaintext highlighter-rouge">http</code> or <code class="language-plaintext highlighter-rouge">socks</code>. Blank is the same as <code class="language-plaintext highlighter-rouge">direct</code>. <h2 id="paging-through-paginating-apis">Paging through paginating APIs</h2> Introduced in release: 1.20. Remote APIs frequently implement some sort of pagination as a way of limiting results. However, if you are performing bulk data analysis, it is necessary to reassemble the data into one larger dataset. Drill’s auto-pagination features allow this to happen in the background, so that the user will get clean data back. To use a paginator, you simply have to configure the paginator in the connection for the particular API. <h3 id="words-of-caution">Words of Caution</h3> While extremely powerful, the auto-pagination feature has the potential to run afoul of APIs rate limits and even potentially DDOS an API. <h3 id="offset-pagination">Offset Pagination</h3> Offset Pagination uses commands similar to SQL which has a <code class="language-plaintext highlighter-rouge">LIMIT</code> and an <code class="language-plaintext highlighter-rouge">OFFSET</code>. With an offset paginator, let’s say you want 200 records and the page size is 50 records, the offset paginator will break up your query into 4 requests as shown below: <ul> <li>myapi.com?limit=50&offset=0</li> <li>myapi.com?limit=50?offset=50</li> <li>myapi.com?limit=50&offset=100</li> <li>myapi.com?limit=50&offset=150</li> </ul> To configure an offset paginator, simply add the following to the configuration for your connection. <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"paginator": { "limitParam": "<limit>", "offsetParam": "<offset>", "pageSize": 100, "method": "OFFSET" } </code></pre></div></div> <h3 id="page-pagination">Page Pagination</h3> Page pagination is very similar to offset pagination except instead of using an <code class="language-plaintext highlighter-rouge">OFFSET</code> it uses a page number. <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "paginator": { "pageParam": "page", "pageSizeParam": "per_page", "pageSize": 100, "method": "PAGE" } </code></pre></div></div> In either case, the <code class="language-plaintext highlighter-rouge">pageSize</code> parameter should be set to the maximum page size allowable by the API. This will minimize the number of requests Drill is making. <h2 id="examples">Examples</h2> <h3 id="example-1-reference-data-a-sunrisesunset-api">Example 1: Reference Data, A Sunrise/Sunset API</h3> The API sunrise-sunset.org returns data in the following format: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code> "results": { "sunrise":"7:27:02 AM", "sunset":"5:05:55 PM", "solar_noon":"12:16:28 PM", "day_length":"9:38:53", "civil_twilight_begin":"6:58:14 AM", "civil_twilight_end":"5:34:43 PM", "nautical_twilight_begin":"6:25:47 AM", "nautical_twilight_end":"6:07:10 PM", "astronomical_twilight_begin":"5:54:14 AM", "astronomical_twilight_end":"6:38:43 PM" }, "status":"OK" } </code></pre></div></div> To query this API, set the configuration as follows: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "type": "http", "cacheResults": false, "enabled": true, "timeout": 5, "connections": { "sunrise": { "url": "https://api.sunrise-sunset.org/json", "requireTail": true, "method": "GET", "headers": null, "authType": "none", "userName": null, "password": null, "postBody": null, "inputType": "json" } } </code></pre></div></div> Then, to execute a query: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT api_results.results.sunrise AS sunrise, api_results.results.sunset AS sunset FROM http.sunrise.`?lat=36.7201600&lng=-4.4203400&date=today` AS api_results; </code></pre></div></div> Which yields the following results: <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>|------------|------------| | sunrise | sunset | |------------|------------| | 7:17:46 AM | 5:01:33 PM | |------------|------------| 1 row selected (0.632 seconds) </code></pre></div></div> <h4 id="using-parameters">Using Parameters</h4> We can improvide the above configuration to use <code class="language-plaintext highlighter-rouge">WHERE</code> clause filters and a <code class="language-plaintext highlighter-rouge">dataPath</code> to skip over the unwanted parts of the message body. Set the configuration as follows: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "type": "http", "cacheResults": false, "enabled": true, "timeout": 5, "connections": { "sunrise": { "url": "https://api.sunrise-sunset.org/json", "requireTail": false, "method": "GET", "dataPath": "results", "headers": null, "params": [ "lat", "lng", "date" ], "authType": "none", "userName": null, "password": null, "postBody": null } } </code></pre></div></div> Then, to execute a query: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT sunrise, sunset FROM http.sunrise WHERE `lat` = 36.7201600 AND `lng` = -4.4203400 AND `date` = 'today' </code></pre></div></div> Which yields the same results as before. <h3 id="example-2-jira">Example 2: JIRA</h3> JIRA Cloud has a REST API which is <a href="https://developer.atlassian.com/cloud/jira/platform/rest/v3/?utm_source=%2Fcloud%2Fjira%2Fplatform%2Frest%2F&utm_medium=302">documented here</a>. To connect Drill to JIRA Cloud, use the following configuration: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "type": "http", "cacheResults": false, "timeout": 5, "connections": { "jira": { "url": "https://<project>.atlassian.net/rest/api/3/", "method": "GET", "dataPath": "issues", "headers": { "Accept": "application/json" }, "authType": "basic", "userName": "<username>", "password": "<API Key>", "postBody": null } }, "enabled": true } </code></pre></div></div> Once you’ve configured Drill to query the API, you can now easily access any of your data in JIRA. The JIRA API returns highly nested data, however with a little preparation, it is pretty straightforward to transform it into a more useful table. For instance, the query below: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT key, t.fields.issueType.name AS issueType, SUBSTR(t.fields.created, 1, 10) AS created, SUBSTR(t.fields.updated, 1, 10) AS updated, t.fields.assignee.displayName as assignee, t.fields.creator.displayName as creator, t.fields.summary AS summary, t.fields.status.name AS currentStatus, t.fields.priority.name AS priority, t.fields.labels AS labels, t.fields.subtasks AS subtasks FROM http.jira.`search?jql=project%20%3D%20<project>&&maxResults=100 AS t` </code></pre></div></div> The query below counts the number of issues by priority: <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SELECT t.fields.priority.name AS priority, COUNT(*) AS issue_count FROM http.jira.`search?jql=project%20%3D%20<project>&&maxResults=100` AS t GROUP BY priority ORDER BY issue_count DESC </code></pre></div></div> <img src="images/docs/issue_count.png" alt="Issue Count by Priority" /> <h2 id="limitations">Limitations</h2> <ol> <li> The plugin is supposed to follow redirects, however if you are using authentication, you may encounter errors or empty responses if you are counting on the endpoint for redirection. </li> <li> At this time, the plugin does not support any authentication other than basic authentication. </li> <li> This plugin does not implement join filter pushdowns (only constant pushdowns are supported). Join pushdown has the potential to improve performance if you use the HTTP service joined to another table. </li> <li> This plugin only reads JSON, CSV and XML response data. </li> <li> <code class="language-plaintext highlighter-rouge">POST</code> bodies can only be in the format of key/value pairs. Some APIs accept JSON based <code class="language-plaintext highlighter-rouge">POST</code> bodies but this is not currently supported. </li> <li> When using <code class="language-plaintext highlighter-rouge">dataPath</code>, the returned message should a single JSON object. The field pointed to by the <code class="language-plaintext highlighter-rouge">dataPath</code> should contain a single JSON object or an array of objects. </li> <li> When not using <code class="language-plaintext highlighter-rouge">dataPath</code>, the response should be a single JSON object, an array of JSON objects, or a series of line-delimited JSON objects (the so-called <a href="http://jsonlines.org/">jsonlines</a> format.) </li> <li> Parameters are considered optional; no error will be given if a query omits parameters. An enhancement would be to mark parameters as required: all are required or just some. If parameters are required, but omitted, the report service will likely return an error. </li> </ol> <h2 id="troubleshooting">Troubleshooting</h2> If anything goes wrong, Drill will provide a detailed error message, including URL: <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DATA_READ ERROR: Failed to read the HTTP response body Error message: Read timed out Connection: sunrise Plugin: api URL: https://api.sunrise-sunset.org/json?lat=36.7201600&lng=-4.4203400&date=today Fragment: 0:0 </code></pre></div></div> If using a “tail” in the query, verify that the tail is quoted using back-ticks as shown in the examples. Check that the URL is correct. If not, check the plugin configuration properties described above to find out why the pieces were assembed as you want. If the query works but delivers unexpected results, check the Drill log file. Drill logs a message like the following at the info level when opening the HTTP connection: <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Connection: sunrise, Method: GET, URL: https://api.sunrise-sunset.org/json?lat=36.7201600&lng=-4.4203400&date=today </code></pre></div></div> If the query runs, but produces odd results, try a simple <code class="language-plaintext highlighter-rouge">SELECT *</code> query. This may reveal if there is unexpected message context in addition to the data. Use the <code class="language-plaintext highlighter-rouge">dataPath</code> property to ignore the extra content. <div class="doc-nav"> <a href="/docs/druid-storage-plugin/">← Druid Storage Plugin</a><a href="/docs/elasticsearch-storage-plugin/">ElasticSearch Storage Plugin →</a> </div> </div> </div> </div> </div> <div id="footer" class="mw"> <div class="wrapper"> Copyright © 2012-2025 The Apache Software Foundation, licensed under the Apache License, Version 2.0. Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners. </div> </div> <script type="text/javascript" src="https://s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> </body> </html>

CINXE.COM

HTTP Storage Plugin - Apache Drill