diff --git a/README.md b/README.md index 5a20cdcc59b..3af18aabb74 100644 --- a/README.md +++ b/README.md @@ -16,60 +16,54 @@ -## 🗄️ Overview +## Overview -Hue is a mature SQL Assistant for querying Databases & Data Warehouses, trusted by: +Hue is an open-source SQL Assistant for querying databases and data warehouses. Use it to explore your data, write SQL with autocomplete, and share the results. -* **1000+ customers** worldwide -* **Top Fortune 500** enterprises +### Key features -Organizations use Hue to quickly answer questions via self-service querying, executing hundreds of thousands of queries daily. +- **Interactive SQL editor** with syntax highlighting and autocomplete. +- **File Browser** to navigate and operate on HDFS, S3, ABFS, Ozone, and Google Storage (GS) files. +- **Job Browser** to monitor and manage Hive queries, Impala queries, YARN applications, and Livy Spark jobs. +- **Table Browser** to explore and manage database tables, schemas, and metadata. +- **Table Importer** to create Hive and Impala tables from CSV and Excel files, uploaded from your computer or imported from a remote file system (HDFS, S3, ABFS, Ozone, GS). +- **Multiple database connectors**, including Hive, Impala, MySQL, and PostgreSQL. -### Key Features +## Useful links -- ✏️ **Interactive SQL editing** with syntax highlighting and autocomplete -- 📁 **File Browser** for navigating and operating on HDFS, S3, ABFS, Ozone, and Google Storage (GS) files -- 📊 **Job Browser** for monitoring and managing Hive queries, Impala queries, YARN applications, and Livy Spark jobs -- 🗃️ **Table Browser** for exploring and managing database tables, schemas, and metadata -- 📤 **Table Importer** for creating Hive and Impala tables from CSV/Excel files, with support for uploading from local system or importing from remote filesystems (HDFS, S3, ABFS, Ozone, GS) -- 🔗 **Multiple database connectors** including Hive, Impala, MySQL, PostgreSQL, and more - -## 🔗 Useful Links - -- 🏠 **Website**: [gethue.com](https://gethue.com) -- 🔌 **Connect to a database**: [Configuration Guide](https://docs.gethue.com/administrator/configuration/connectors/) -- 🛠️ **Build your own Editor**: [SQL Scratchpad](https://docs.gethue.com/developer/components/scratchpad/) -- ⚙️ **Query Service**: [Kubernetes Setup](https://docs.gethue.com/administrator/installation/cloud/#kubernetes) and [API](https://docs.gethue.com/developer/api/) +- **Website**: [gethue.com](https://gethue.com) +- **Documentation**: [docs.gethue.com](https://docs.gethue.com) +- **Connect to a database**: [Connectors](https://docs.gethue.com/administrator/configuration/connectors/) +- **Build your own editor**: [SQL Scratchpad](https://docs.gethue.com/developer/components/scratchpad/) +- **Query service**: [Kubernetes setup](https://docs.gethue.com/administrator/installation/cloud/#kubernetes) and [API](https://docs.gethue.com/developer/api/) ![Hue Editor](https://cdn.gethue.com/uploads/2021/02/hue-4.9.png) -## 🚀 Getting Started - -### Try Hue Now +## Get started -- 🌐 **Live demo**: [demo.gethue.com](https://demo.gethue.com/) -- 🧪 **Quick integrations**: +- **Live demo**: [demo.gethue.com](https://demo.gethue.com/) +- **Quick integrations**: - [Hive](https://docs.gethue.com/administrator/configuration/connectors/#apache-hive) - [Impala](https://gethue.com/blog/quickstart-sql-editor-for-apache-impala/) - - [Trino/Presto](https://docs.gethue.com/administrator/configuration/connectors/#trino-presto) + - [Trino/Presto](https://docs.gethue.com/administrator/configuration/connectors/#trino) - [Phoenix SQL / HBase](https://gethue.com/blog/querying-live-kafka-data-in-apache-hbase-with-phoenix/) - [Spark SQL](https://gethue.com/blog/querying-spark-sql-with-spark-thrift-server-and-hue-editor/) -Choose one of these deployment options to start the server, then [configure the databases](https://docs.gethue.com/administrator/configuration/connectors/) you want to query: +Choose a deployment option to start the server. Then [configure the databases](https://docs.gethue.com/administrator/configuration/connectors/) you want to query. -### 🐳 Docker +### Docker -Start Hue instantly: +Start Hue: ```bash docker run -it -p 8888:8888 gethue/hue:latest ``` -Hue will be available at [http://localhost:8888](http://localhost:8888) +Open [http://localhost:8888](http://localhost:8888). The first user to log in becomes the superuser. -📖 See the [Docker Guide](https://github.com/cloudera/hue/tree/master/tools/docker/hue) or watch the [Quick Start Video](http://gethue.com/getting-started-with-hue-in-2-minutes-with-docker/) +> **Note:** The image uses an embedded SQLite database by default. Connect Hue to MySQL or PostgreSQL before real use. See the [Docker guide](tools/docker/hue/README.md). -### ☸️ Kubernetes +### Kubernetes ```bash helm repo add gethue https://helm.gethue.com @@ -77,44 +71,44 @@ helm repo update helm install hue gethue/hue ``` -📖 Read more about configurations in the [Kubernetes docs](tools/kubernetes/) +See the [Kubernetes guide](tools/kubernetes/README.md) and the [Hue Helm chart](tools/kubernetes/helm/hue/README.md) for the configuration options. -### 💻 Development Setup +### Build from source -#### Quick Start with Docker +Hue requires Python 3.11 and Node.js 22.13 or later. -Use the [Dev Environment Docker](https://docs.gethue.com/developer/development/#dev-docker) for the fastest setup. +1. Install the [dependencies](https://docs.gethue.com/administrator/installation/dependencies/) for your operating system. +2. Clone the repository and build Hue: -#### Manual Setup + ```bash + git clone https://github.com/cloudera/hue.git + cd hue + export PYTHON_VER=python3.11 + make apps + ``` -1. Install [dependencies](https://docs.gethue.com/administrator/installation/dependencies/) -2. Clone and build: - -```bash -git clone https://github.com/cloudera/hue.git -cd hue -make apps -build/env/bin/hue runserver -``` +3. Start the development server: -Hue will be available at [http://localhost:8000](http://localhost:8000) + ```bash + build/env/bin/hue runserver + ``` -📖 Read more in the [development documentation](https://docs.gethue.com/developer/development/) +Open [http://localhost:8000](http://localhost:8000). -## 🧩 Components +For a development environment in Docker, see [Dev Docker](https://docs.gethue.com/developer/development/#dev-docker). For more details, see the [development documentation](https://docs.gethue.com/developer/development/). -Hue offers several powerful components: +## Components -- **SQL Editor** - Interactive query interface -- **SQL Parsers** - Syntax handling for multiple dialects -- **REST/Python/CLI APIs** - Programmatic access to all functionality +- **SQL Editor**: interactive query interface. +- **SQL Parsers**: syntax handling for multiple dialects. +- **REST and Python APIs**: programmatic access to Hue. -📖 Learn about [components](https://docs.gethue.com/developer/components/) and [APIs](https://docs.gethue.com/developer/api/) +Learn more about the [components](https://docs.gethue.com/developer/components/) and the [APIs](https://docs.gethue.com/developer/api/). -## 🤝 Contributing +## Contributing -We welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) guide to get started. +Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) to get started. -## 📜 License +## License [Apache License, Version 2.0](LICENSE.txt) diff --git a/apps/beeswax/src/beeswax/management/commands/beeswax_install_examples.py b/apps/beeswax/src/beeswax/management/commands/beeswax_install_examples.py index 990e63d7db9..8434b6360a8 100644 --- a/apps/beeswax/src/beeswax/management/commands/beeswax_install_examples.py +++ b/apps/beeswax/src/beeswax/management/commands/beeswax_install_examples.py @@ -15,13 +15,13 @@ # See the License for the specific language governing permissions and # limitations under the License. -import os import csv -import pwd import json import logging +import os +import pwd -from django.core.management.base import BaseCommand +from django.core.management.base import BaseCommand, CommandError from django.utils.translation import gettext as _ from beeswax.conf import LOCAL_EXAMPLES_DATA_DIR @@ -31,8 +31,8 @@ from desktop.lib.exceptions_renderable import PopupException from desktop.models import Document2, Document2Permission from hadoop import cluster -from notebook.models import MockRequest, _get_example_directory, import_saved_beeswax_query, make_notebook -from useradmin.models import User, get_default_user_group, install_sample_user +from notebook.models import _get_example_directory, import_saved_beeswax_query, make_notebook, MockRequest +from useradmin.models import get_default_user_group, install_sample_user, User LOG = logging.getLogger() @@ -42,9 +42,13 @@ class InstallException(Exception): class Command(BaseCommand): - args = ' ' help = 'Install examples but do not overwrite them.' + def add_arguments(self, parser): + # From the command line: beeswax_install_examples [db_name]. Callers in the code use handle() with options. + parser.add_argument('args', nargs='*', metavar='dialect [db_name]', + help='The dialect (hive or impala) and the database (default is "default").') + def handle(self, *args, **options): if args: dialect = args[0] @@ -55,6 +59,8 @@ def handle(self, *args, **options): self.tables = None interpreter = None else: + if 'dialect' not in options: + raise CommandError('Expected arguments: [db_name]') dialect = options['dialect'] db_name = options.get('db_name', 'default') interpreter = options.get('interpreter') # Only when connectors are enabled. Later will deprecate `dialect`. @@ -288,7 +294,7 @@ def load_partition(self, django_user, partition_spec, filepath, columns): partition_dir = self._get_partition_dir(partition_spec) hdfs_root_destination = self._get_hdfs_root_destination(django_user, subdir=partition_dir) filename = filepath.split('/')[-1] - hdfs_file_destination = self._upload_to_hdfs(django_user, filepath, hdfs_root_destination, filename) + self._upload_to_hdfs(django_user, filepath, hdfs_root_destination, filename) hql = LOAD_PARTITION_HQL % {'tablename': self.name, 'partition_spec': partition_spec, 'filepath': hdfs_root_destination} diff --git a/apps/beeswax/src/beeswax/management/commands/close_commands_tests.py b/apps/beeswax/src/beeswax/management/commands/close_commands_tests.py new file mode 100644 index 00000000000..5281c63bc11 --- /dev/null +++ b/apps/beeswax/src/beeswax/management/commands/close_commands_tests.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python +# Licensed to Cloudera, Inc. under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. Cloudera, Inc. licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from io import StringIO +from unittest.mock import MagicMock, patch + +import pytest +from django.core.management import call_command, CommandError + +from beeswax.management.commands import close_queries, close_sessions + +COMMANDS = [ + ('close_queries', close_queries.Command), + ('close_sessions', close_sessions.Command), +] + + +@pytest.mark.parametrize('name, command_class', COMMANDS) +class TestCloseCommandsArguments: + def parse(self, name, command_class, argv): + return command_class().create_parser('hue', name).parse_args(argv) + + def test_defaults_to_7_days_and_no_scope(self, name, command_class): + options = self.parse(name, command_class, []) + + assert options.age_in_days == 7 + assert options.scope is None + + def test_age_in_days(self, name, command_class): + options = self.parse(name, command_class, ['0']) + + assert options.age_in_days == 0 + assert options.scope is None + + def test_age_in_days_and_all(self, name, command_class): + options = self.parse(name, command_class, ['3', 'all']) + + assert options.age_in_days == 3 + assert options.scope == 'all' + + def test_rejects_an_unknown_scope(self, name, command_class): + with pytest.raises(CommandError): + self.parse(name, command_class, ['3', 'everything']) + + def test_rejects_a_non_numeric_age(self, name, command_class): + with pytest.raises(CommandError): + self.parse(name, command_class, ['week']) + + +class TestCloseCommandsExecution: + def run_command(self, name, module, model, *args): + model_objects = MagicMock() + model_objects.filter.return_value = model_objects + model_objects.all.return_value = model_objects + model_objects.__iter__.return_value = iter([]) + model_objects.__len__.return_value = 0 + out = StringIO() + + with patch.object(module, model) as model_class, patch.dict('os.environ', {'HIVE_CONF_DIR': '/tmp'}): + model_class.objects = model_objects + model_class.STATE = MagicMock() + with patch('beeswax.hive_site.reset'), patch('beeswax.hive_site.get_conf'), patch('beeswax.conf.HIVE_CONF_DIR'): + call_command(name, *args, stdout=out) + + return out.getvalue() + + def test_close_queries_with_the_age_and_all(self): + output = self.run_command('close_queries', close_queries, 'QueryHistory', '0', 'all') + + assert 'Closing (all=True) HiveServer2 queries older than 0 days' in output + + def test_close_queries_without_arguments(self): + output = self.run_command('close_queries', close_queries, 'QueryHistory') + + assert 'Closing (all=False) HiveServer2 queries older than 7 days' in output + + def test_close_sessions_with_the_age_and_all(self): + output = self.run_command('close_sessions', close_sessions, 'Session', '2', 'all') + + assert 'Closing (all=True) HiveServer2 sessions older than 2 days' in output + + def test_close_sessions_without_arguments(self): + output = self.run_command('close_sessions', close_sessions, 'Session') + + assert 'Closing (all=False) HiveServer2 sessions older than 7 days' in output diff --git a/apps/beeswax/src/beeswax/management/commands/close_queries.py b/apps/beeswax/src/beeswax/management/commands/close_queries.py index 38d7f39e5b1..028eb80ff3a 100644 --- a/apps/beeswax/src/beeswax/management/commands/close_queries.py +++ b/apps/beeswax/src/beeswax/management/commands/close_queries.py @@ -15,16 +15,15 @@ # See the License for the specific language governing permissions and # limitations under the License. -from builtins import str import logging -from datetime import datetime, timedelta +from builtins import str +from datetime import datetime, timedelta from django.core.management.base import BaseCommand from beeswax.models import QueryHistory from beeswax.server import dbms - LOG = logging.getLogger() @@ -37,16 +36,25 @@ class Command(BaseCommand): Closing (all=True) queries older than 7 days... 0 queries closed. """ - args = ' (default is 7)' help = 'Close finished Hive queries older than 7 days.' + def add_arguments(self, parser): + parser.add_argument('age_in_days', nargs='?', type=int, default=7, + help='Close the queries older than this number of days (default is 7).') + parser.add_argument('scope', nargs='?', choices=['all'], + help="Pass 'all' to close the queries whatever their state, not only the finished ones.") + def handle(self, *args, **options): - days = int(args[0]) if len(args) >= 1 else 7 - close_all = args[1] == 'all' if len(args) >= 2 else False + days = options['age_in_days'] + close_all = options['scope'] == 'all' self.stdout.write('Closing (all=%s) HiveServer2 queries older than %s days...\n' % (close_all, days)) - queries = QueryHistory.objects.filter(last_state__in=[QueryHistory.STATE.expired.value, QueryHistory.STATE.failed.value, QueryHistory.STATE.available.value]) + queries = QueryHistory.objects.filter(last_state__in=[ + QueryHistory.STATE.expired.value, + QueryHistory.STATE.failed.value, + QueryHistory.STATE.available.value, + ]) if close_all: queries = QueryHistory.objects.all() @@ -54,8 +62,8 @@ def handle(self, *args, **options): queries = queries.filter(submission_date__lte=datetime.today() - timedelta(days=days)) import os + import beeswax - from beeswax import conf from beeswax import hive_site try: beeswax.conf.HIVE_CONF_DIR.set_for_testing(os.environ['HIVE_CONF_DIR']) diff --git a/apps/beeswax/src/beeswax/management/commands/close_sessions.py b/apps/beeswax/src/beeswax/management/commands/close_sessions.py index 7797efdc13f..eadc2568a78 100644 --- a/apps/beeswax/src/beeswax/management/commands/close_sessions.py +++ b/apps/beeswax/src/beeswax/management/commands/close_sessions.py @@ -15,16 +15,15 @@ # See the License for the specific language governing permissions and # limitations under the License. -from builtins import str import logging -from datetime import datetime, timedelta +from builtins import str +from datetime import datetime, timedelta from django.core.management.base import BaseCommand from beeswax.models import Session from beeswax.server import dbms - LOG = logging.getLogger() @@ -37,12 +36,17 @@ class Command(BaseCommand): Closing (all=True) queries older than 7 days... 0 queries closed. """ - args = ' (default is 7)' help = 'Close finished Hive queries older than 7 days.' + def add_arguments(self, parser): + parser.add_argument('age_in_days', nargs='?', type=int, default=7, + help='Close the sessions older than this number of days (default is 7).') + parser.add_argument('scope', nargs='?', choices=['all'], + help="Pass 'all' to close the sessions of all the applications, not only of Beeswax.") + def handle(self, *args, **options): - days = int(args[0]) if len(args) >= 1 else 7 - close_all = args[1] == 'all' if len(args) >= 2 else False + days = options['age_in_days'] + close_all = options['scope'] == 'all' self.stdout.write('Closing (all=%s) HiveServer2 sessions older than %s days...\n' % (close_all, days)) @@ -56,6 +60,7 @@ def handle(self, *args, **options): self.stdout.write('Found %d open HiveServer2 sessions to close' % len(sessions)) import os + import beeswax from beeswax import hive_site diff --git a/apps/hbase/src/hbase/management/commands/hbase_setup.py b/apps/hbase/src/hbase/management/commands/hbase_setup.py index 5be56beb755..aa717bc423d 100644 --- a/apps/hbase/src/hbase/management/commands/hbase_setup.py +++ b/apps/hbase/src/hbase/management/commands/hbase_setup.py @@ -24,19 +24,21 @@ from desktop.lib.paths import get_apps_root from hbase.api import HbaseApi -from useradmin.models import install_sample_user +from useradmin.models import install_sample_user, User LOG = logging.getLogger() class Command(BaseCommand): help = 'Create and fill some demo tables in the first configured cluster.' - args = '' + + def add_arguments(self, parser): + parser.add_argument('args', nargs='*', metavar='username', help='The user that creates the tables (default is the sample user).') def handle(self, *args, **options): if args: - user = args[0] - elif options and options['user']: + user = User.objects.get(username=args[0]) + elif options.get('user'): user = options['user'] else: user = install_sample_user() diff --git a/desktop/conf.dist/hue.ini b/desktop/conf.dist/hue.ini index df60cca0f1b..50bb9f335b3 100644 --- a/desktop/conf.dist/hue.ini +++ b/desktop/conf.dist/hue.ini @@ -61,14 +61,13 @@ http_500_debug_mode=false ## django_email_backend=django.core.mail.backends.smtp.EmailBackend # Set to true to use CherryPy as the webserver, set to false -# to use Gunicorn as the webserver. Defaults to CherryPy if -# key is not specified. -## use_cherrypy_server=true +# to use Gunicorn as the webserver. Defaults to false (Gunicorn) on Python 3. +## use_cherrypy_server=false -# Gunicorn work class: gevent or evenlet, gthread or sync. -## gunicorn_work_class=eventlet +# Gunicorn work class: gevent, eventlet, gthread or sync. +## gunicorn_work_class=gthread -# The number of Gunicorn worker processes. If not specified, it uses: (number of CPU * 2) + 1. +# The number of Gunicorn worker processes. The default is 1. ## gunicorn_number_of_workers=1 # Workers silent for more than this many seconds are killed and restarted. @@ -91,11 +90,12 @@ http_500_debug_mode=false # This should be the hadoop cluster admin ## default_hdfs_superuser=hdfs -# If set to false, runcpserver will not actually start the web server. -# Used if Apache is being used as a WSGI container. -## enable_server=yes +# If set to false, the CherryPy web server (use_cherrypy_server=true) will not actually start. +# Used if Apache is being used as a WSGI container. Not used by the Gunicorn web server. +## enable_server=true -# Number of threads used by the CherryPy web server +# Number of threads used by the web server (Gunicorn or CherryPy). Also sizes the connection pools +# used to talk to the other services. ## cherrypy_server_threads=50 # This property specifies the maximum size of the receive buffer in bytes in thrift sasl communication, @@ -114,7 +114,7 @@ http_500_debug_mode=false # Flag to disable webpage caching. Enabling this flag will reduce the performance of the application but it ensures that # the client is always receiving the latest version of the resource. -## custom_cache_control=true +## custom_cache_control=false # Enable TLS 1.3 support when available. Requires OpenSSL 1.1.1+ and Python 3.7+ ## ssl_tls13_enabled=true @@ -150,7 +150,7 @@ http_500_debug_mode=false ## secure_content_security_policy="script-src 'self' 'unsafe-inline' 'unsafe-eval' *.googletagmanager.com *.doubleclick.net data:;img-src 'self' *.doubleclick.net http://*.tile.osm.org *.tile.osm.org *.gstatic.com data:;style-src 'self' 'unsafe-inline' fonts.googleapis.com;connect-src 'self' *.google-analytics.com;frame-src *;child-src 'self' data: *.vimeo.com;object-src 'none'" # Enable nonce attribute to remove unsafe-inline and auto remove unsafe-inline from csp -## csp_nonce=true +## csp_nonce=false # Strict-Transport-Security HTTP Strict Transport Security(HSTS) is a policy which is communicated by the server to the user agent via HTTP response header field name "Strict-Transport-Security". HSTS policy specifies a period of time during which the user agent(browser) should only access the server in a secure fashion(https). ## secure_ssl_redirect=False @@ -206,17 +206,27 @@ http_500_debug_mode=false ## http_x_frame_options=SAMEORIGIN # Enable X-Forwarded-Host header if the load balancer requires it. -## use_x_forwarded_host=true +# Defaults to true when hue_load_balancer is set, false otherwise. +## use_x_forwarded_host=false # Enable X-Forwarded-For header if hive/impala requires it. -## enable_xff_for_hive_impala=true +## enable_xff_for_hive_impala=false # Enable X-CSRF-Token header if hive/impala requries it -## enable_x_csrf_token_for_hive_impala=true +## enable_x_csrf_token_for_hive_impala=false # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false +# Enable or disable Cross-Origin Resource Sharing (CORS). +## cors_enabled=true + +# This value determines whether the server allows cookies in the cross-site HTTP requests. +## cors_allow_credentials=true + +# A comma separated list of origins allowed for CORS. +## cors_allowed_origins= + # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend @@ -241,7 +251,10 @@ http_500_debug_mode=false # Id of the cluster where Hue is located. ## cluster_id='default' -# Choose whether to show the new SQL editor. +# Number of user access to remember per view per user. +## user_access_history_size=10 + +# Deprecated: choose whether to show the new SQL editor. ## use_new_editor=true # Global setting to allow or disable end user downloads in all Hue. @@ -261,9 +274,30 @@ http_500_debug_mode=false # Choose whether to use new charting library across the whole Hue. ## use_new_charts=false +# Choose whether to show the new Assist panel or the old. +## use_new_assist_panel=false + +# Feature flag to enable new Hue Storage browser. +## enable_new_storage_browser=false + +# Feature flag to enable new Hue Importer. +## enable_new_importer=false + # Choose whether to allow multi tenancy or not. ## enable_organizations=false +# Turn on the Connector configuration and usage. +## enable_connectors=false + +# Comma separated list of connector types to hide. +## connectors_blacklist= + +# If not empty, comma separated list of connector types to keep. +## connectors_whitelist= + +# Custom URL to use for the dashboard application and editor result charting +## custom_dashboard_url= + # Choose whether the editor autocomplete should gather suggestions from external source or not. The editor # autocomplete uses various sources for its suggestions, listing databases, tables, columns files etc. The results are # cached on the client (see cacheable_ttl) so the calls are kept to a minimum but if you prefer to disable these calls @@ -424,7 +458,7 @@ http_500_debug_mode=false ## api_auth=rest_framework_simplejwt.authentication.JWTAuthentication,rest_framework.authentication.SessionAuthentication # Class which defines extra accessor methods for User objects. -## user_aug=desktop.auth.backend.DefaultUserAugmentor +## user_augmentor=desktop.auth.backend.DefaultUserAugmentor # The service to use when querying PAM. ## pam_service=login @@ -491,6 +525,10 @@ idle_session_timeout=-1 # If behind_reverse_proxy is True, it will look for the IP address from this header. Default: HTTP_X_FORWARDED_FOR ## reverse_proxy_header=HTTP_X_FORWARDED_FOR +# If True, will auto log any request as a `hue` user that needs to exist. +# Only use this for development or when Hue sits behind a trusted authenticating proxy. +## auto_login_enabled=false + [[[jwt]]] # Endpoint to fetch the public key from verification server. # Also adds custom JWT Authentication backend for REST APIs in top priority if set. @@ -610,7 +648,7 @@ idle_session_timeout=-1 ## group_name_attr=cn # The attribute of the group object which identifies the members of the group -## group_member_attr=members +## group_member_attr=member [[[ldap_servers]]] @@ -771,7 +809,10 @@ idle_session_timeout=-1 ## http_only=true # Use session-length cookies. Logs out the user when she closes the browser window. -## expire_at_browser_close=false +## expire_at_browser_close=true + +# CSRF cookie age in seconds. If the value is set to 0, it means per session. +## csrf_cookie_age=0 # If set, limits the number of concurrent user sessions. 1 represents 1 browser session per user. Default: 0 (unlimited sessions per user) ## concurrent_user_session_limit=0 @@ -816,7 +857,10 @@ tls=no # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Frequency in seconds with which Hue will renew its keytab -## REINIT_FREQUENCY=3600 +## reinit_frequency=3600 + +# kt-renewer sleep timeout in seconds. +## kt_renewer_timeout=10 # Path to keep Kerberos credentials cached ## ccache_path=/var/run/hue/hue_krb5_ccache # Path to kinit @@ -870,7 +914,7 @@ tls=no ## oidc_op_jwks_endpoint=https://keycloak.example.com/auth/realms/Cloudera/protocol/openid-connect/certs # Whether Hue as OpenID Connect client verify SSL cert -## oidc_verify_ssl=true +## oidc_verify_ssl=false # OIDC authentication request extra params ## oidc_auth_request_extra_params={} @@ -951,6 +995,9 @@ tls=no # Number of query results rows to fetch into the result storage. ## fetch_result_limit=2000 +# Django file cache class to use to temporarily store query results +## result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://localhost:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}' + # Django file storage class to use to temporarily store query results ## result_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "./logs"}}' @@ -994,9 +1041,15 @@ tls=no # Switch on the integration with the Task Scheduler. ## beat_enabled=False +# Path to a file containing a list of beat schedules. +# beat_schedules_file= + # Number of query results rows to fetch into the result storage. ## fetch_result_limit=2000 +# Django file cache class to use to temporarily store query results +## result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://localhost:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}' + # Django file storage class to use to temporarily store query results ## result_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "./logs"}}' @@ -1170,8 +1223,12 @@ tls=no # enable_custom_connectors=false ## Flag to enable the bulk submission of queries as a background task through Oozie. +## Defaults to true when the Oozie app is loaded (not in app_blacklist), false otherwise. # enable_batch_execute=true +## Flag to turn on the built-in hints on Impala queries in the editor. +# enable_query_analysis=false + ## Flag to enable all interpreters (Hive and Impala are added by default) related to every whitelisted app. # enable_all_interpreters=false @@ -1400,15 +1457,15 @@ tls=no # Activate the SQL Dashboard (beta). ## has_sql_enabled=false -# Activate the Query Builder (beta). -## has_query_builder_enabled=false - # Activate the static report layout (beta). ## has_report_enabled=false # Activate the new grid layout system. ## use_gridster=true +# Activate the simplified drag in the dashboard. +## use_new_add_method=false + # Activate the widget filter and comparison (beta). ## has_widget_filter=false @@ -1422,11 +1479,11 @@ tls=no # [[[solr]]] # Requires Solr 6+ -## analytics=true +## analytics=false ## nesting=false # [[[sql]]] -## analytics=true +## analytics=false ## nesting=false @@ -1466,8 +1523,8 @@ fs_defaultfs=hdfs://localhost:8020 # Whether Hue should list this HDFS cluster. For historical reason there is no way to disable HDFS. ## is_enabled=true -# Size, in bytes, of the chunks Django should store into memory and feed into the handler. Default is 64MB. -## upload_chunk_size=64*1024*1024 +# Size, in bytes, of the chunks Django should store into memory and feed into the handler. Default is 128MB. +## upload_chunk_size=128*1024*1024 # Configuration for YARN (MR2) # ------------------------------------------------------------------------ @@ -1553,7 +1610,7 @@ submit_to=True # Alternatively, use Service Discovery for LLAP (Hive Server Interactive) and/or Hiveserver2, this will override server and thrift port # Whether to use Service Discovery for LLAP -## hive_discovery_llap = true +## hive_discovery_llap = false # is llap (hive server interactive) running in an HA configuration (more than 1) # important as the zookeeper structure is different @@ -1566,7 +1623,7 @@ submit_to=True ## hive_discovery_llap_znode = /hiveserver2-hive2 # Whether to use Service Discovery for HiveServer2 -## hive_discovery_hs2 = true +## hive_discovery_hs2 = false # Hiveserver2 is hive-site hive.server2.zookeeper.namespace ex hiveserver2 = /hiverserver2 ## hive_discovery_hiveserver2_znode = /hiveserver2 @@ -1599,7 +1656,8 @@ submit_to=True ## use_get_log_api=false # Limit the number of partitions that can be listed. -## list_partitions_limit=10000 +# A positive value will be set as the LIMIT. +## list_partitions_limit=1000 # The maximum number of partitions that will be included in the SELECT * LIMIT sample query for partitioned tables. ## query_partitions_limit=10 @@ -1608,6 +1666,14 @@ submit_to=True # A value of -1 means there will be no limit. ## download_row_limit=100000 +# Deprecated: a limit to the number of cells (rows * columns) that can be downloaded from a query +# (e.g. 10K rows * 1K columns = 10M cells). A value of -1 means there will be no limit. +# When download_row_limit is not set, it defaults to this value divided by 100. +## download_cell_limit=10000000 + +# The max number of records in the result set permitted to apply a natural sort to the database or tables list. +## apply_natural_sort_max=2000 + # A limit to the number of bytes that can be downloaded from a query before it is truncated. # A value of -1 means there will be no limit. ## download_bytes_limit=-1 @@ -1619,7 +1685,7 @@ submit_to=True # Hue will use at most this many HiveServer2 sessions per user at a time. # For Tez, increase the number to more if you need more than one query at the time, e.g. 2 or 3 (Tez has a maximum of 1 query by session). # -1 is unlimited number of sessions. -## max_number_of_sessions=1 +## max_number_of_sessions=-1 # When set to True, Hue will close sessions created for background queries and open new ones as needed. # When set to False, Hue will keep sessions created for background queries opened and reuse them as needed. @@ -1829,7 +1895,13 @@ submit_to=True [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. -## archive_upload_tempdir=/tmp +## archive_upload_tempdir=/tmp/hue_uploads + +# Configure chunk size of the chunked file uploader. Default chunk size is set to 5MB. +## file_upload_chunk_size=5242880 + +# Configure the maximum number of concurrent connections (chunks) for file uploads using the chunked file uploader. +## concurrent_max_connections=5 # Show Download Button for HDFS file browser. ## show_download_button=true @@ -1986,7 +2058,7 @@ submit_to=True ## enable_kafka=false # Flag to turn on the direct upload of a small file. -## enable_direct_upload=true +## enable_direct_upload=false ########################################################################### @@ -2029,6 +2101,9 @@ submit_to=True # Show the Hive section for listing the query history and providing more troubleshooting information. ## enable_hive_query_browser=false +# Show the version 2 of job/query History which unifies the all into one. +## enable_history_v2=false + # Use the proxy API instead of the ORM to access the query_store. ## use_proxy=true @@ -2518,7 +2593,7 @@ submit_to=True # Limits found entities to a specific cluster. When empty the entities from all clusters will be included in the # search results. -## search_cluster= +## search_cluster=cm # Set to true when authenticating via kerberos instead of username/password ## kerberos_enabled=core_site.is_kerberos_enabled() @@ -2544,7 +2619,7 @@ submit_to=True # Username of the LDAP user used for authentication. ## navmetadataserver_ldap_user=hue # LDAP password of the user used for authentication. -## navmetadataserver_ldap_ppassword= +## navmetadataserver_ldap_password= # Execute this script to produce the LDAP password. This will be used when the plain password is not set. ## navmetadataserver_ldap_password_script= diff --git a/desktop/conf/pseudo-distributed.ini.tmpl b/desktop/conf/pseudo-distributed.ini.tmpl index 11cd9ed4689..af9a72dc142 100644 --- a/desktop/conf/pseudo-distributed.ini.tmpl +++ b/desktop/conf/pseudo-distributed.ini.tmpl @@ -66,14 +66,13 @@ django_email_backend=django.core.mail.backends.console.EmailBackend # Set to true to use CherryPy as the webserver, set to false - # to use Gunicorn as the webserver. Defaults to CherryPy if - # key is not specified. - ## use_cherrypy_server=true + # to use Gunicorn as the webserver. Defaults to false (Gunicorn) on Python 3. + ## use_cherrypy_server=false - # Gunicorn work class: gevent or evenlet, gthread or sync. - ## gunicorn_work_class=eventlet + # Gunicorn work class: gevent, eventlet, gthread or sync. + ## gunicorn_work_class=gthread - # The number of Gunicorn worker processes. If not specified, it uses: (number of CPU * 2) + 1. + # The number of Gunicorn worker processes. The default is 1. ## gunicorn_number_of_workers=1 # Workers silent for more than this many seconds are killed and restarted. @@ -96,11 +95,12 @@ # This should be the hadoop cluster admin ## default_hdfs_superuser=hdfs - # If set to false, runcpserver will not actually start the web server. - # Used if Apache is being used as a WSGI container. - ## enable_server=yes + # If set to false, the CherryPy web server (use_cherrypy_server=true) will not actually start. + # Used if Apache is being used as a WSGI container. Not used by the Gunicorn web server. + ## enable_server=true - # Number of threads used by the CherryPy web server + # Number of threads used by the web server (Gunicorn or CherryPy). Also sizes the connection pools + # used to talk to the other services. ## cherrypy_server_threads=50 # This property specifies the maximum size of the receive buffer in bytes in thrift sasl communication, @@ -119,7 +119,7 @@ # Flag to disable webpage caching. Enabling this flag will reduce the performance of the application but it ensures # that the client is always receiving the latest version of the resource. - ## custom_cache_control=true + ## custom_cache_control=false # Enable TLS 1.3 support when available. Requires OpenSSL 1.1.1+ and Python 3.7+ ## ssl_tls13_enabled=true @@ -155,7 +155,7 @@ ## secure_content_security_policy="script-src 'self' 'unsafe-inline' 'unsafe-eval' *.googletagmanager.com *.doubleclick.net data:;img-src 'self' *.doubleclick.net http://*.tile.osm.org *.tile.osm.org *.gstatic.com data:;style-src 'self' 'unsafe-inline' fonts.googleapis.com;connect-src 'self' *.google-analytics.com;frame-src *;child-src 'self' data: *.vimeo.com;object-src 'none'" # Enable nonce attribute to remove unsafe-inline and auto remove unsafe-inline from csp - ## csp_nonce=true + ## csp_nonce=false # Strict-Transport-Security HTTP Strict Transport Security(HSTS) is a policy which is communicated by the server to the user agent via HTTP response header field name "Strict-Transport-Security". HSTS policy specifies a period of time during which the user agent(browser) should only access the server in a secure fashion(https). @@ -212,17 +212,27 @@ ## http_x_frame_options=SAMEORIGIN # Enable X-Forwarded-Host header if the load balancer requires it. - ## use_x_forwarded_host=true + # Defaults to true when hue_load_balancer is set, false otherwise. + ## use_x_forwarded_host=false # Enable X-Forwarded-For header if hive/impala requires it. - ## enable_xff_for_hive_impala=true + ## enable_xff_for_hive_impala=false # Enable X-CSRF-Token header if hive/impala requires it. - ## enable_x_csrf_token_for_hive_impala=true + ## enable_x_csrf_token_for_hive_impala=false # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER. ## secure_proxy_ssl_header=false + # Enable or disable Cross-Origin Resource Sharing (CORS). + ## cors_enabled=true + + # This value determines whether the server allows cookies in the cross-site HTTP requests. + ## cors_allow_credentials=true + + # A comma separated list of origins allowed for CORS. + ## cors_allowed_origins= + # Comma-separated list of Django middleware classes to use. # See https://docs.djangoproject.com/en/1.4/ref/middleware/ for more details on middlewares in Django. ## middleware=desktop.auth.backend.LdapSynchronizationBackend @@ -247,7 +257,10 @@ # Id of the cluster where Hue is located. ## cluster_id='default' - # Choose whether to show the new SQL editor. + # Number of user access to remember per view per user. + ## user_access_history_size=10 + + # Deprecated: choose whether to show the new SQL editor. ## use_new_editor=true # Global setting to allow or disable end user downloads in all Hue. @@ -267,6 +280,15 @@ # Choose whether to use new charting library across the whole Hue. ## use_new_charts=false + # Choose whether to show the new Assist panel or the old. + ## use_new_assist_panel=false + + # Feature flag to enable new Hue Storage browser. + ## enable_new_storage_browser=false + + # Feature flag to enable new Hue Importer. + ## enable_new_importer=false + # Use storage connector system for multi-cloud object storage access. # Default: true. When enabled, replaces legacy AWS_ACCOUNTS with modern storage connector configuration. # Set to false to use legacy AWS_ACCOUNTS instead. @@ -275,6 +297,18 @@ # Choose whether to allow multi tenancy or not. ## enable_organizations=false + # Turn on the Connector configuration and usage. + ## enable_connectors=false + + # Comma separated list of connector types to hide. + ## connectors_blacklist= + + # If not empty, comma separated list of connector types to keep. + ## connectors_whitelist= + + # Custom URL to use for the dashboard application and editor result charting + ## custom_dashboard_url= + # Choose whether the editor autocomplete should gather suggestions from external source or not. The editor # autocomplete uses various sources for its suggestions, listing databases, tables, columns files etc. The results are # cached on the client (see cacheable_ttl) so the calls are kept to a minimum but if you prefer to disable these calls @@ -430,7 +464,7 @@ ## api_auth=rest_framework_simplejwt.authentication.JWTAuthentication,rest_framework.authentication.SessionAuthentication # Class which defines extra accessor methods for User objects. - ## user_aug=desktop.auth.backend.DefaultUserAugmentor + ## user_augmentor=desktop.auth.backend.DefaultUserAugmentor # The service to use when querying PAM. ## pam_service=login @@ -497,6 +531,10 @@ # If behind_reverse_proxy is True, it will look for the IP address from this header. Default: HTTP_X_FORWARDED_FOR ## reverse_proxy_header=HTTP_X_FORWARDED_FOR + # If True, will auto log any request as a `hue` user that needs to exist. + # Only use this for development or when Hue sits behind a trusted authenticating proxy. + ## auto_login_enabled=false + [[[jwt]]] # Endpoint to fetch the public key from verification server. # Also adds custom JWT Authentication backend for REST APIs in top priority if set. @@ -514,6 +552,12 @@ # Verify custom JWT signature. ## verify=true + # Choose whether Hue should validate the certificate presented by the JWT key server. + # Disabling this allows an attacker able to intercept the connection to supply their own + # signing keys, so only turn it off against a trusted endpoint with a self-signed certificate. + # Defaults to the global ssl_validate setting. + ## ssl_cert_ca_verify=true + # Configuration options for connecting to LDAP and Active Directory # ------------------------------------------------------------------- [[ldap]] @@ -610,7 +654,7 @@ ## group_name_attr=cn # The attribute of the group object which identifies the members of the group - ## group_member_attr=members + ## group_member_attr=member [[[ldap_servers]]] @@ -743,6 +787,9 @@ ## name=desktop/desktop.db ## options={} + # Database schema, to be used only when public schema is revoked in postgres + ## schema=public + # Configuration options for specifying the Desktop session. # For more info, see https://docs.djangoproject.com/en/1.11/topics/http/sessions/ # ------------------------------------------------------------------------ @@ -767,7 +814,10 @@ ## http_only=true # Use session-length cookies. Logs out the user when she closes the browser window. - ## expire_at_browser_close=false + ## expire_at_browser_close=true + + # CSRF cookie age in seconds. If the value is set to 0, it means per session. + ## csrf_cookie_age=0 # If set, limits the number of concurrent user sessions. 1 represents 1 browser session per user. Default: 0 (unlimited sessions per user) ## concurrent_user_session_limit=0 @@ -812,7 +862,10 @@ # Kerberos principal name for Hue ## hue_principal=hue/hostname.foo.com # Frequency in seconds with which Hue will renew its keytab - ## REINIT_FREQUENCY=3600 + ## reinit_frequency=3600 + + # kt-renewer sleep timeout in seconds. + ## kt_renewer_timeout=10 # Path to keep Kerberos credentials cached ## ccache_path=/var/run/hue/hue_krb5_ccache # Path to kinit @@ -866,7 +919,7 @@ ## oidc_op_jwks_endpoint=https://keycloak.example.com/auth/realms/Cloudera/protocol/openid-connect/certs # Whether Hue as OpenID Connect client verify SSL cert - ## oidc_verify_ssl=true + ## oidc_verify_ssl=false # OIDC authentication request extra params ## oidc_auth_request_extra_params={} @@ -944,9 +997,15 @@ # Switch on the integration with the Task Scheduler. ## beat_enabled=False + # Path to a file containing a list of beat schedules. + # beat_schedules_file= + # Number of query results rows to fetch into the result storage. ## fetch_result_limit=2000 + # Django file cache class to use to temporarily store query results + ## result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://localhost:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}' + # Django file storage class to use to temporarily store query results ## result_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "./logs"}}' @@ -972,9 +1031,15 @@ # Switch on the integration with the Task Scheduler. ## beat_enabled=False + # Path to a file containing a list of beat schedules. + # beat_schedules_file= + # Number of query results rows to fetch into the result storage. ## fetch_result_limit=2000 + # Django file cache class to use to temporarily store query results + ## result_cache='{"BACKEND": "django_redis.cache.RedisCache", "LOCATION": "redis://localhost:6379/0", "OPTIONS": {"CLIENT_CLASS": "django_redis.client.DefaultClient"},"KEY_PREFIX": "queries"}' + # Django file storage class to use to temporarily store query results ## result_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "./logs"}}' @@ -1149,7 +1214,11 @@ # enable_custom_connectors=false ## Flag to enable the bulk submission of queries as a background task through Oozie. - # enable_batch_execute=false + ## Defaults to true when the Oozie app is loaded (not in app_blacklist), false otherwise. + # enable_batch_execute=true + + ## Flag to turn on the built-in hints on Impala queries in the editor. + # enable_query_analysis=false ## Flag to enable all interpreters (Hive and Impala are added by default) related to every whitelisted app. # enable_all_interpreters=false @@ -1378,15 +1447,15 @@ # Activate the SQL Dashboard (beta). ## has_sql_enabled=false - # Activate the Query Builder (beta). - ## has_query_builder_enabled=false - # Activate the static report layout (beta). ## has_report_enabled=false # Activate the new grid layout system. ## use_gridster=true + # Activate the simplified drag in the dashboard. + ## use_new_add_method=false + # Activate the widget filter and comparison (beta). ## has_widget_filter=false @@ -1400,11 +1469,11 @@ # [[[solr]]] # Requires Solr 6+ - ## analytics=true + ## analytics=false ## nesting=false # [[[sql]]] - ## analytics=true + ## analytics=false ## nesting=false @@ -1444,8 +1513,8 @@ # Whether Hue should list this HDFS cluster. For historical reason there is no way to disable HDFS. ## is_enabled=true - # Size, in bytes, of the chunks Django should store into memory and feed into the handler. Default is 64MB. - ## upload_chunk_size=64*1024*1024 + # Size, in bytes, of the chunks Django should store into memory and feed into the handler. Default is 128MB. + ## upload_chunk_size=128*1024*1024 # Configuration for YARN (MR2) # ------------------------------------------------------------------------ @@ -1531,7 +1600,7 @@ # Alternatively, use Service Discovery for LLAP (Hive Server Interactive) and/or Hiveserver2, this will override server and thrift port # Whether to use Service Discovery for LLAP - ## hive_discovery_llap = true + ## hive_discovery_llap = false # is llap (hive server interactive) running in an HA configuration (more than 1) # important as the zookeeper structure is different @@ -1544,7 +1613,7 @@ ## hive_discovery_llap_znode = /hiveserver2-hive2 # Whether to use Service Discovery for HiveServer2 - ## hive_discovery_hs2 = true + ## hive_discovery_hs2 = false # Hiveserver2 is hive-site hive.server2.zookeeper.namespace ex hiveserver2 = /hiverserver2 ## hive_discovery_hiveserver2_znode = /hiveserver2 @@ -1577,7 +1646,8 @@ ## use_get_log_api=false # Limit the number of partitions that can be listed. - ## list_partitions_limit=10000 + # A positive value will be set as the LIMIT. + ## list_partitions_limit=1000 # The maximum number of partitions that will be included in the SELECT * LIMIT sample query for partitioned tables. ## query_partitions_limit=10 @@ -1586,6 +1656,14 @@ # A value of -1 means there will be no limit. ## download_row_limit=100000 + # Deprecated: a limit to the number of cells (rows * columns) that can be downloaded from a query + # (e.g. 10K rows * 1K columns = 10M cells). A value of -1 means there will be no limit. + # When download_row_limit is not set, it defaults to this value divided by 100. + ## download_cell_limit=10000000 + + # The max number of records in the result set permitted to apply a natural sort to the database or tables list. + ## apply_natural_sort_max=2000 + # A limit to the number of bytes that can be downloaded from a query before it is truncated. # A value of -1 means there will be no limit. ## download_bytes_limit=-1 @@ -1597,7 +1675,7 @@ # Hue will use at most this many HiveServer2 sessions per user at a time. # For Tez, increase the number to more if you need more than one query at the time, e.g. 2 or 3 (Tez has a maximum of 1 query by session). # -1 is unlimited number of sessions. - ## max_number_of_sessions=1 + ## max_number_of_sessions=-1 # When set to True, Hue will close sessions created for background queries and open new ones as needed. # When set to False, Hue will keep sessions created for background queries opened and reuse them as needed. @@ -1805,7 +1883,13 @@ [filebrowser] # Location on local filesystem where the uploaded archives are temporary stored. - ## archive_upload_tempdir=/tmp + ## archive_upload_tempdir=/tmp/hue_uploads + + # Configure chunk size of the chunked file uploader. Default chunk size is set to 5MB. + ## file_upload_chunk_size=5242880 + + # Configure the maximum number of concurrent connections (chunks) for file uploads using the chunked file uploader. + ## concurrent_max_connections=5 # Show Download Button for HDFS file browser. ## show_download_button=true @@ -1962,7 +2046,7 @@ ## enable_kafka=false # Flag to turn on the direct upload of a small file. - ## enable_direct_upload=true + ## enable_direct_upload=false ########################################################################### @@ -2005,6 +2089,9 @@ # Show the Hive section for listing the query history and providing more troubleshooting information. ## enable_hive_query_browser=false + # Show the version 2 of job/query History which unifies the all into one. + ## enable_history_v2=false + # Use the proxy API instead of the ORM to access the query_store. ## use_proxy=true @@ -2270,9 +2357,6 @@ # https://docs.djangoproject.com/en/1.4/ref/databases/ ## options={} - # Database schema, to be used only when public schema is revoked in postgres - ## schema=public - ########################################################################### # Settings to configure SAML ########################################################################### @@ -2497,7 +2581,7 @@ # Limits found entities to a specific cluster. When empty the entities from all clusters will be included in the # search results. - ## search_cluster= + ## search_cluster=cm # Set to true when authenticating via kerberos instead of username/password ## kerberos_enabled=core_site.is_kerberos_enabled() @@ -2523,7 +2607,7 @@ # Username of the LDAP user used for authentication. ## navmetadataserver_ldap_user=hue # LDAP password of the user used for authentication. - ## navmetadataserver_ldap_ppassword= + ## navmetadataserver_ldap_password= # Execute this script to produce the LDAP password. This will be used when the plain password is not set. ## navmetadataserver_ldap_password_script= diff --git a/desktop/core/src/desktop/management/commands/create_proxy_app.py b/desktop/core/src/desktop/management/commands/create_proxy_app.py index d019d6aaf8e..f7cec7bae81 100644 --- a/desktop/core/src/desktop/management/commands/create_proxy_app.py +++ b/desktop/core/src/desktop/management/commands/create_proxy_app.py @@ -16,10 +16,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +import logging import os import re import shutil -import logging from django.core.management.base import BaseCommand, CommandError from django.utils.translation import gettext as _ @@ -30,9 +30,12 @@ class Command(BaseCommand): help = _("Creates a Hue proxy application directory structure.") - args = "app_name app_web_url [app_dir]" label = _('application name') + def add_arguments(self, parser): + parser.add_argument('args', nargs='*', metavar='app_name [app_web_url] [app_dir]', + help='The application name, its web URL and the directory where to create it (default is the current one).') + def handle(self, *args, **options): if len(args) > 3 or len(args) == 0: raise CommandError(_("Expected arguments: app_name app_web_url [app_dir]")) diff --git a/desktop/core/src/desktop/management/commands/positional_arguments_tests.py b/desktop/core/src/desktop/management/commands/positional_arguments_tests.py new file mode 100644 index 00000000000..32fd3f25754 --- /dev/null +++ b/desktop/core/src/desktop/management/commands/positional_arguments_tests.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python +# Licensed to Cloudera, Inc. under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. Cloudera, Inc. licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Django only accepts the positional arguments that a command declares, so a command that reads them from `args` +has to declare them in `add_arguments`. +""" + +from unittest.mock import patch + +import pytest +from django.core.management import call_command, CommandError + +from beeswax.management.commands import beeswax_install_examples +from desktop.management.commands import create_proxy_app +from hbase.management.commands import hbase_setup + +COMMANDS = [ + ('beeswax_install_examples', beeswax_install_examples, [('hive',), ('impala', 'sample_db')]), + ('hbase_setup', hbase_setup, [(), ('bob',)]), + ('create_proxy_app', create_proxy_app, [('my_app',), ('my_app', 'http://localhost:8080'), ('my_app', 'http://localhost:8080', '/tmp')]), +] + + +@pytest.mark.parametrize('name, module, calls', COMMANDS, ids=[command[0] for command in COMMANDS]) +def test_the_positional_arguments_are_passed_to_handle(name, module, calls): + for positional_arguments in calls: + with patch.object(module.Command, 'handle', return_value=None) as handle: + call_command(name, *positional_arguments) + + assert handle.call_args.args == positional_arguments + + +def test_beeswax_install_examples_requires_a_dialect_from_the_command_line(): + with pytest.raises(CommandError, match='Expected arguments'): + call_command('beeswax_install_examples') diff --git a/docs/README.md b/docs/README.md index 7ad3d5f5b91..f6861634978 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,5 @@ -# Documentations +# Documentation -docs.gethue.com: [source](/docs/docs-site), [documentation](https://docs.gethue.com/developer/development/#blog--website) - -gethue.com: [source](/docs/gethue), [documentation](https://docs.gethue.com/developer/development/#documentation) - -Designs: [source](/docs/designs/) +- docs.gethue.com: [source](docs-site), [how to build it](https://docs.gethue.com/developer/development/#documentation-1) +- gethue.com: [source](gethue), [how to build it](https://docs.gethue.com/developer/development/#gethuecom-blog--website) +- Designs: [source](designs/) diff --git a/docs/designs/hue5.md b/docs/designs/hue5.md deleted file mode 100644 index 7a1b1ead228..00000000000 --- a/docs/designs/hue5.md +++ /dev/null @@ -1,193 +0,0 @@ -# Compose - -A modernized and pluggable Query Editor Service. - -Compose is the name of the new packages (pypi and npm) that will be made available via Hue 5. The goal is to have traditional Hue repository import these packages (note: API/UI core logic refactoring will happen first the other way around before "filling -up" the compose API). Hue is 11 year old and saw many technically revolutions and technical shifts across the year. Hue saw 4 iterative rewrites so far and this is now #5. - -The API revamp is coupled with the UI revamp, Editor 2 and Connectors. - -Overall repo architecture: - - gethue/compose - compose - editor - connectors - iam (auth, users, orgs..) - documents - [scheduler, catalog, storage ..] - docs - tools, docker, docker-compose, kubernetes... - compose-ui - -A single approach like in Hue 4 instead of a multi repo like https://github.com/kubernetes projects (which has docs, demos, roadmap... repos) was taken to keep things simple/contained for now. - -Note that a lot of the format/syntax will change until v1 and in the future the low level source of truth will be in the code itself while this document will stay high level. - -Here are the core components: - -* api server -* static UI -* DBs - -And the optional services: - -* Task Server -* Task Scheduler -* Web Socket Server -* Caching Server - -## API - -A refactored compose module (very similar to the simple REST API) is now available for easily handling the query operations. Those used to be mingled in multiple parts in Hue 4 (e.g. models, API, views, beeswax...). - -High level example: - - from compose.editor import execute - - execute('SELECT 1', url="sqlite:///../db.sqlite3") - execute('SELECT 1', {"url: "", ...}) - execute('SELECT 1', connector_id) - - execute('SELECT 1', {connector}, sync=True) - -## Ops - -The system will be one-click to run and provide standard Cloud CNCF monitoring to allow a transparent view of the activity of the Query Service. - -* docker/docker compose of the full system -* kubernetes -* basic analytics, metrics, tracing, logs - -# REST API - -Clean, skeletons for modern. Leveraging Django REST Framework. -Runs on port 8005. - -## Execute sync query - - curl -u hue:hue -X POST http://localhost:8000/notebook/api/execute/sqlite --data 'snippet={"statement":"SELECT 1000, 1001"}' - -## Auth - -CORS is enabled -No CSRF needed with JWT - -Ask for JWT token: - - curl -X POST -d "username=hue&password=hue" http://localhost:8000/api-token-auth/ - - curl -X POST http://localhost:8000/query/ --data 'snippet={"statement":"SELECT 1000, 1001"}' -H "Authorization: JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoxLCJ1c2VybmFtZSI6Imh1ZSIsImV4cCI6MTYxMjk3MTc0MywiZW1haWwiOiJodWVAZ2V0aHVlLmNvbSIsIm9yaWdfaWF0IjoxNjEyODg1MzQzfQ._HViX-D9h1ZfcXPAaY4KL0SNkx7MvXCH41T8Upkja3o" | jq - -Basic Auth without token: - - curl -u hue:hue - -## UI Integration - -A live demo with the SQL Scratchpad is coming. In the meantime: - - - - - -## Pypi - -Comes via https://pypi.org/project/gethue/ - -After installing it: - - python3 -m pip install gethue - -Either use the Python compose API or boot the REST API: - - gunicorn core.wsgi --bind 0.0.0.0:8005 - -## API Live Docs - - http://127.0.0.1:8000/api/schema/swagger-ui/ - http://127.0.0.1:8000/api/schema/re-doc/ - -# Dev - -There is no more makefile and the OS dependencies can be setup via a one time bash script: - - ./install.sh - -Above script also create the virtual environment and install all the Python module dependencies from [hue/requirements.txt](hue/requirements.txt). - -Afterwards all the dependencies are contained into the dedicated virtual environment: - - source python_env/bin/activate - -Starting the API server is now a standard Django command: - - python manage.py runserver - -## Docker - - docker build hue -t gethue/compose-api:latest -f hue/docker/Dockerfile - docker push gethue/compose-api:latest - - docker run -it -p 8005:8000 gethue/compose-api:latest - - curl -X POST http://localhost:8000/notebook/api/execute/sqlite --data 'snippet={"statement":"SELECT 1000, 1001"}' | jq - -## CI - -The CI is now based on GitHub actions to keeps all the things in one place. Direct pushes to master branch are disabled and need to go via the pre-commit CI. - -Various actions are happening in order to check the styles and run the tests. e.g. - - black - isort - coverage - license check - unused imports removal - ... - -[pre-commits](https://pre-commit.com/hooks.html) hooks can be run automatically - -# Hue 5 vs Hue 4 differences - -There is a 80% drop of custom/code debt while keeping 80% of the functionalities. Main focus is on the Editor while Storage Browser, Catalog can be simply brought back gradually and quickly. - -* This is now Django3/Vue.js3/Python3 only -* The old apps, desktop/core, desktop/libs are now at the same level (many old dead libs/code like ZooKeeper, Oozie, etc are not ported) -* SQL Thrift interface is phased out in favor or SqlAlchemy (the former saw Beeswax, HiverServer and HiveServer2 transition shifts) -* There is no more cumbersome 'ext-py' directory, now using [hue/requirements.txt](hue/requirements.txt). -* CI is via GitHub actions vs CircleCI -* CI now automatically perform the grunts work/checks for coding styles, missing licenses, unused imports etc... -* There is no more custom Hue apps, command, packages, permissions... but only standard Django -* ini was phased out in favor or the regular Django settings.py -* Templates (Django, Mako) are not used anymore and the UI is not 100% separated and based on View.js. -* Hue is now released as a standard pypi module instead of a custom tarball that required to be compiled again. -* nose was phased out in favor of pytest for running the tests -* code coverage is now made transparent and visible on each PR -* Jira is not used anymore in favor of GitHub issues and projects -* Old websites are not present anymore - -Notes: -* requirements.txt/setup.py needs to be split with regard to dev/prod versions -* no full dependencies on all the modules: e.g. LDAP, mysql DB, phoenix... are optional and easy to add/remove -* internationalization still need to be ported somehow diff --git a/docs/docs-site/README.md b/docs/docs-site/README.md index cc5c9f80e6e..87bf1cc3f39 100644 --- a/docs/docs-site/README.md +++ b/docs/docs-site/README.md @@ -1,22 +1,34 @@ -# Docs Site +# Docs site -## Update GetHue version +The source of [docs.gethue.com](https://docs.gethue.com), built with [Hugo](https://gohugo.io). -Docs depends on gethue NPM package for live demos. Kindly follow these steps to update gethue to the version of your choice. +## Update the GetHue version + +The docs depend on the `gethue` npm package for the live demos. Follow these steps to update it. + +### Prerequisites + +- Node.js 22.13 or later +- A shell in the `docs/docs-site` directory ### 1. Install GetHue - # x.x.x - Version you are updating to - npm install --dev-save gethue@x.x.x +Replace `x.x.x` with the version you want to use: + +```bash +npm install --save-dev gethue@x.x.x +``` -### 2. Update static resources +### 2. Update the static resources - npm run update-static +```bash +npm run update-static +``` -This must update the resources used in the docs. On `git status` you must see modifications on `./package.json`, `./package-lock.json` & `static/js/gethue/`. +`git status` shows changes in `package.json`, `package-lock.json`, and `static/js/gethue/`. -On `hugo serve` you must see the updated resources in the docs UI. +Run `hugo serve` and check that the docs UI uses the updated resources. -### 3. Commit changes +### 3. Commit the changes -If you want to share the updated version with everyone, commit the changed files. +To share the new version, commit the changed files. diff --git a/docs/docs-site/content/_index.md b/docs/docs-site/content/_index.md index 1b91232fffd..443148cec26 100644 --- a/docs/docs-site/content/_index.md +++ b/docs/docs-site/content/_index.md @@ -7,18 +7,12 @@ pre = "1. " +++ ![Hue Logo Back-end](/images/hue_logo.png) -

Documentation

+Hue is an open-source SQL Assistant for querying databases and data warehouses. -Hue is a mature SQL Assistant for querying Databases & Data Warehouses. +Read more on [gethue.com](https://gethue.com) and: -* 1000+ customers -* Top Fortune 500 - -use Hue to quickly answer questions via self-service querying and are executing 100s of 1000s of queries daily. - -Read more on [gethue.com](http://gethue.com) and -- Connect to a [Database](/administrator/configuration/connectors/) -- Build your own Editor with the [SQL Scratchpad](/developer/components/scratchpad/) -- Or Query Service by leveraging the [REST API](/developer/api/rest/) and [Kubernetes](/administrator/installation/cloud/#kubernetes) +- Connect to a [database](/administrator/configuration/connectors/) +- Build your own editor with the [SQL Scratchpad](/developer/components/scratchpad/) +- Run a query service with the [REST API](/developer/api/rest/) and [Kubernetes](/administrator/installation/cloud/#kubernetes) !["Hue Editor"](https://cdn.gethue.com/uploads/2021/02/hue-4.9.png) diff --git a/docs/docs-site/content/administrator/_index.md b/docs/docs-site/content/administrator/_index.md index 35704e4627a..e870bbbf484 100644 --- a/docs/docs-site/content/administrator/_index.md +++ b/docs/docs-site/content/administrator/_index.md @@ -6,11 +6,8 @@ chapter = false pre = "1. " +++ -### Guide - -The following instructions describe how to: +This guide describes how to: * [Install](/administrator/installation/) and [configure](/administrator/configuration/server/) the server -* Point to external services to query (e.g. Databases, Hadoop, S3, ...) via [connectors](/administrator/configuration/connectors/) - -Then find more details about [administrating](/administrator/administration/reference/) the service. +* Query external services (for example databases, Hadoop, and S3) with [connectors](/administrator/configuration/connectors/) +* [Administer](/administrator/administration/reference/) the service diff --git a/docs/docs-site/content/administrator/administration/_index.md b/docs/docs-site/content/administrator/administration/_index.md index 7f09d0c3e40..bcd72be3600 100644 --- a/docs/docs-site/content/administrator/administration/_index.md +++ b/docs/docs-site/content/administrator/administration/_index.md @@ -5,4 +5,4 @@ draft: false weight: 4 --- -This sections details the "ops" part of Hue, e.g. how to monitor the servers, get a big picture of the architecture, manage users and permissions. +This section describes the operations side of Hue: the architecture and sizing, how to monitor the servers, manage the database, and manage users and permissions. diff --git a/docs/docs-site/content/administrator/administration/database.md b/docs/docs-site/content/administrator/administration/database.md index c43d2e0ef0c..d9f193e6d25 100644 --- a/docs/docs-site/content/administrator/administration/database.md +++ b/docs/docs-site/content/administrator/administration/database.md @@ -5,263 +5,304 @@ draft: false weight: 4 --- -Hue requires a SQL database to store small amounts of data, including user account information as well as history of queries and sharing permissions. +Hue stores a small amount of data in a SQL database: user accounts, query history, saved documents, and sharing permissions. -By default, Hue is configured to use an embedded SQLite database so that it starts but many errors will come up due to the lack of transactions. +By default, Hue uses an embedded SQLite database so that it starts without setup. SQLite does not support concurrent writes well, so you can get `database is locked` errors. Use another database for anything beyond a test. -This section contains instructions for configuring Hue with another database. +This page describes how to configure Hue with another database and how to operate it. -## Configuring with another Database +## Prerequisites -Although SQLite is the default database type, some advanced users may prefer -to have Hue access an alternate database type. Note that if you elect to -configure Hue to use an external database, upgrades may require more manual -steps in the future. +* A running MySQL, MariaDB, PostgreSQL, or Oracle server. +* The Python client library of your database installed in the Hue virtual environment. See [Database dependencies](/administrator/installation/dependencies/#database). +* The path to your `hue.ini`. Hue loads all the `.ini` files of the `desktop/conf` directory (or of the `HUE_CONF_DIR` directory), and later files in alphabetical order take precedence. -The following instructions are for MySQL, though you can also configure Hue to work with other common databases such as PostgreSQL and Oracle. +> **Note:** If you use an external database, upgrades can require more manual steps. +The settings of the `[[database]]` section of the `hue.ini` are: -### MySQL - -To configure Hue to store data in MySQL: - -1. Create a new database in MySQL and grant privileges to a Hue user to manage - this database. - - mysql> create database hue; - Query OK, 1 row affected (0.01 sec) - mysql> grant all on hue.* to 'hue'@'localhost' identified by 'secretpassword'; - Query OK, 0 rows affected (0.00 sec) +| Setting | Description | +|---|---| +| `engine` | `mysql` (also for MariaDB), `postgresql_psycopg2`, `oracle`, or `sqlite3` (default) | +| `host` | Database server host | +| `port` | Database server port | +| `user` | Database user | +| `password` | Database password. Alternatively, set `password_script` to a script that prints it. | +| `name` | Database name. For `sqlite3`, the path of the file. | +| `options` | Extra connection options | +| `conn_max_age` | Seconds to keep connections open (default `0`) | -2. Shut down Hue if it is running. +Run `build/env/bin/hue config_help | less` to list all the options. -3. To migrate your existing data to MySQL, use the following command to dump the - existing database data to a text file. Note that using the ".json" extension - is required. +## Configure another database - /usr/share/hue/build/env/bin/hue dumpdata > $some-temporary-file.json +The steps below use `/usr/share/hue` as the Hue installation directory. Replace it with yours. -4. Open the `hue.ini` file in a text editor. Directly below the - `[[database]]` line, add the following options (and modify accordingly for - your MySQL setup): - - host=localhost - port=3306 - engine=mysql - user=hue - password=secretpassword - name=hue +### MySQL -5. As the Hue user, configure Hue to load the existing data and create the database tables: +1. Create a database and a user in MySQL: - /usr/share/hue/build/env/bin/hue migrate - mysql -uhue -psecretpassword -e "DELETE FROM hue.django_content_type;" - /usr/share/hue/build/env/bin/hue loaddata $temporary-file-containing-dumped-data.json + ```sql + CREATE DATABASE hue; + CREATE USER 'hue'@'localhost' IDENTIFIED BY 'secretpassword'; + GRANT ALL ON hue.* TO 'hue'@'localhost'; + ``` -Your system is now configured and you can start the Hue server as normal. +2. Stop Hue if it is running. -### PostgreSQL +3. To keep your existing data, dump it to a file. The `.json` extension is required: -Log on to PostgreSQL: - psql -h localhost -U hue -d hue + ```bash + /usr/share/hue/build/env/bin/hue dumpdata > /tmp/hue_database_dump.json + ``` -Password for user hue: +4. Open the `hue.ini` file in a text editor. Under `[desktop]`, add the options below directly after the `[[database]]` line, and modify them for your setup: -Drop the foreign key constraint from auth_permission: + ```ini + [desktop] + [[database]] + host=localhost + port=3306 + engine=mysql + user=hue + password=secretpassword + name=hue + ``` - \d auth_permission; - ALTER TABLE auth_permission DROP CONSTRAINT content_type_id_refs_id_; +5. As the Hue user, create the database tables: -Delete the contents of django_content_type: + ```bash + /usr/share/hue/build/env/bin/hue migrate + ``` - TRUNCATE django_content_type CASCADE; +6. If you dumped your data, remove the content types created by the migration, then load the dump: -Load Database. + ```bash + mysql -uhue -p -e "DELETE FROM hue.django_content_type;" + /usr/share/hue/build/env/bin/hue loaddata /tmp/hue_database_dump.json + ``` - Add the foreign key, content_type_id, to auth_permission: - ALTER TABLE auth_permission ADD FOREIGN KEY (content_type_id) REFERENCES django_content_type(id) DEFERRABLE INITIALLY DEFERRED; +You can now start the Hue server as usual. -### Oracle +### PostgreSQL -Oracle users should delete all content from the Oracle tables after synchronizing and before loading: +1. Create a database and a user: -Log on to Oracle: + ```sql + CREATE USER hue WITH PASSWORD 'secretpassword'; + CREATE DATABASE hue OWNER hue; + ``` - su - oracle - sqlplus / as sysdba +2. Set the `[[database]]` section of the `hue.ini`: -Grant a quota to the tablespace where tables are created (the default is SYSTEM). For example: + ```ini + [desktop] + [[database]] + host=localhost + port=5432 + engine=postgresql_psycopg2 + user=hue + password=secretpassword + name=hue + ``` - ALTER USER hue quota 100m on system; +3. Stop Hue, dump your data as in the MySQL steps, then create the tables: -Log on as the hue: + ```bash + /usr/share/hue/build/env/bin/hue migrate + ``` - sqlplus hue/ +4. To load a dump, log on to PostgreSQL and drop the foreign key constraint from `auth_permission`: -Create a spool script that creates a delete script to clean the content of all tables. + ```bash + psql -h localhost -U hue -d hue + ``` - vi spool_statements.ddl + ```sql + \d auth_permission; + ALTER TABLE auth_permission DROP CONSTRAINT content_type_id_refs_id_; + ``` -Save in spool_statements.ddl (which generates delete_from_tables.ddl) +5. Delete the contents of `django_content_type`: - spool delete_from_tables.ddl - set pagesize 100; - SELECT 'DELETE FROM ' || table_name || ';' FROM user_tables; - commit; - spool off - quit + ```sql + TRUNCATE django_content_type CASCADE; + ``` -Run both scripts: +6. Load the data, then add the foreign key back: - -- Create delete_from_tables.ddl - sqlplus hue/ < spool_statements.ddl + ```bash + /usr/share/hue/build/env/bin/hue loaddata /tmp/hue_database_dump.json + ``` - -- Run delete_from_tables.ddl - sqlplus hue/ < delete_from_tables.ddl + ```sql + ALTER TABLE auth_permission ADD FOREIGN KEY (content_type_id) REFERENCES django_content_type(id) DEFERRABLE INITIALLY DEFERRED; + ``` -#### RAC +### Oracle -The current setup of Oracle is this way: +To migrate data to Oracle, delete all content from the Oracle tables after running `migrate` and before loading the data: - Hostname : ORACLE IP - Database Type: Oracle - Database Name: - Username: - Password: +Log on to Oracle: -This would be an alternate way to address the Oracle RAC issue: +```bash +su - oracle +sqlplus / as sysdba +``` -1. Go to Hue > Configuration > Database > Hue Database Name -2. Enter the database name in the following format -Hue Database Name=(DESCRIPTION=(LOAD_BALANCE=off)(FAILOVER=on)(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3)(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=))(CONNECT_DATA=(SERVICE_NAME=))) +Grant a quota to the tablespace where tables are created (the default is SYSTEM). For example: -If the above does not work, then use the below config info `hue_safety_valve_server` for database: +```sql +ALTER USER hue quota 100m on system; +``` - engine=oracle - port=0 - user= - password= - name=(DESCRIPTION=(LOAD_BALANCE=off)(FAILOVER=on)(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3)(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=))(CONNECT_DATA=(SERVICE_NAME=))) +Log on as the hue: -## Operation +```bash +sqlplus hue/ +``` -### Inspecting +Create a spool script that creates a delete script to clean the content of all tables. -You can inspect your database via its standard own Shell, Hue [SQL Editor](/administrator/configuration/connectors/#databases) itself, or from the command line using the db shell: +```bash +vi spool_statements.ddl +``` - /usr/share/hue/build/env/bin/hue dbshell +Save in spool_statements.ddl (which generates delete_from_tables.ddl) -It is strongly recommended that you avoid making any modifications to the database directly, though this trick can be useful for management or troubleshooting. +```sql +spool delete_from_tables.ddl +set pagesize 100; +SELECT 'DELETE FROM ' || table_name || ';' FROM user_tables; +commit; +spool off +quit +``` -### Backing up +Run both scripts: -It is recommended that you back it up on a regular schedule, and also that you back it up before any upgrade to a new version or doing a data change in the database. +```text +-- Create delete_from_tables.ddl +sqlplus hue/ < spool_statements.ddl -Backing up can be done the standard way of the chosen database or via dumping the records similarly to doing a [migration](#migrating). +-- Run delete_from_tables.ddl +sqlplus hue/ < delete_from_tables.ddl +``` -### Clean up +#### RAC -When the database has too many entries in certain tables, it will cause performance issue. Now Hue config check will help superuser to find this issue. Login as superuser and go to "Administration, this sample screenshot will be displayed in the quick start wizard when the tables have too many entries: +Set the database in the `[[database]]` section of the `hue.ini`: -[][1] +```ini +[desktop] +[[database]] +engine=oracle +host= +port= +user= +password= +name= +``` -Run following clean up command: +You can also use the Oracle service name with `host` and `port` set and `name=:/`. - cd /opt/cloudera/parcels/CDH/lib/hue # Hue home directory - ./build/env/bin/hue desktop_document_cleanup +If this does not work for your RAC cluster, set `port=0` and use a full connect descriptor as the name: -### Migrating +```ini +[desktop] +[[database]] +engine=oracle +port=0 +user= +password= +name=(DESCRIPTION=(LOAD_BALANCE=off)(FAILOVER=on)(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3)(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=))(CONNECT_DATA=(SERVICE_NAME=))) +``` -Hue Custom Databases includes database-specific pages on how to migrate from an old to a new database (e.g. sqlite to Oracle or MySQL). This page summarizes across supported database types. +Hue sets `options={"threaded": True}` by default for Oracle, which is required in a multithreaded environment. -When you change Hue databases, you can migrate the existing data to your new database. +## Operation -#### Dump Database +### Inspect the database -Ensure that the Hue service is stopped and dump the database content like explained in the top section and the `dumpdata` command. +Use the standard shell of your database, the Hue [SQL Editor](/administrator/configuration/connectors/#databases), or the Django database shell: -Edit `/tmp/hue_database_dump.json` by removing all objects with useradmin.userprofile in the model field. +```bash +/usr/share/hue/build/env/bin/hue dbshell +``` -For example: +> **Warning:** Avoid modifying the database directly. Use direct access only for management or troubleshooting. - grep -c useradmin.userprofile /tmp/hue_database_dump.json +### Back up the database - vi /tmp/hue_database_dump.json - { - "pk": 1, - "model": "useradmin.userprofile", - "fields": { - "last_activity": "2016-10-03T10:06:13", - "creation_method": "HUE", - "first_login": false, - "user": 1, - "home_directory": "/user/admin" - } - }, +Back up the database on a regular schedule, and before any upgrade or data change. Use the standard backup tools of your database, or dump the records as described in [Migrate to another database](#migrate-to-another-database). -#### Connect to the new Database +### Clean up the database -Set the appropriate database parameters : +Too many entries in some tables, such as documents, cause performance issues. The Hue configuration check warns superusers about this. Log in as a superuser and open **Hue Administration** to see it. - Hue Database Type: MySQL or PostgreSQL or Oracle - Hue Database Hostname: - Hue Database Port: 3306 or 5432 or 1521 - Hue Database Username: - Hue Database Password: - Hue Database Name: +![Config check for too many documents](https://cdn.gethue.com/uploads/2019/03/Doc2CountCheck.png) -Oracle users only should add support for a multithreaded environment: +To clean up old entries, go to the Hue installation directory and run: -Add support for a multithreaded environment by setting in the hue.ini: +```bash +./build/env/bin/hue desktop_document_cleanup +``` - [desktop] - [[database]] - options={"threaded":True} +### Migrate to another database -In the Hue Web UI, click the home icon Hue Home icon to ensure that all documents were migrated. +You can migrate the existing data when you change databases, for example from SQLite to MySQL or PostgreSQL. - MariaDB / MySQL - Synchronize Database in Cloudera Manager. - Log on to MySQL: - mysql -u root -p - Enter password: +1. Stop Hue and dump the data with `dumpdata`, as in the [MySQL](#mysql) steps. +2. Edit the dump file and remove all the objects whose `model` is `useradmin.userprofile`. For example: -Drop the foreign key constraint from the hue.auth_permission table: + ```bash + grep -c useradmin.userprofile /tmp/hue_database_dump.json + ``` -Execute the following statement to find the `content_type_id_refs_id_` in the CONSTRAINT clause of the CREATE TABLE statement for the `hue.auth_permission` table: + ```json + { + "pk": 1, + "model": "useradmin.userprofile", + "fields": { + "last_activity": "2016-10-03T10:06:13", + "creation_method": "HUE", + "first_login": false, + "user": 1, + "home_directory": "/user/admin" + } + }, + ``` - SHOW CREATE TABLE hue.auth_permission; +3. Configure the new database in the `hue.ini` as described above, and run `migrate`. +4. Load the data as described for your database. +5. Start Hue and check on the Hue home page that all the documents were migrated. -This SHOW CREATE TABLE statement produces output similar to the following: +#### MySQL foreign key - | auth_permission | CREATE TABLE 'auth_permission' ( - 'id' int(11) NOT NULL AUTO-INCREMENT, - 'name' varchar(50) NOT NULL, - 'content_type_id' int(11) NOT NULL, - 'CODENAME' VARCHAR(100) NOT NULL, - PRIMARY KEY ('id'), - UNIQUE KEY 'content_type_id' ('content_type_id', 'codename'), - KEY 'auth_permission_37ef4eb4' ('content_type_id'), - CONSTRAINT 'content_type_id_refs_id_d043b34a' FOREIGN KEY ('content_type_id') - REFERENCES 'django_content_type' ('id') - ) ENGINE=InnoDB AUTO_INCREMENT=229 DEFAULT CHARSET=utf8 | +If loading the data fails on MySQL, drop the foreign key constraint from `hue.auth_permission` before the load and add it back after. -Then execute the following statement to drop the foreign key constraint: +1. Find the constraint name: - ALTER TABLE hue.auth_permission DROP FOREIGN KEY - content_type_id_refs_id_; + ```sql + SHOW CREATE TABLE hue.auth_permission; + ``` -For example, if you used the above output from the SHOW CREATE TABLE statement, you would use the following ALTER TABLE statement: + In the output, find the name in the `CONSTRAINT` clause, for example `content_type_id_refs_id_d043b34a`. - ALTER TABLE hue.auth_permission DROP FOREIGN KEY - content_type_id_refs_id_d043b34a; +2. Drop the constraint, using your constraint name: -Delete the contents of django_content_type: + ```sql + ALTER TABLE hue.auth_permission DROP FOREIGN KEY content_type_id_refs_id_d043b34a; + ``` - DELETE FROM hue.django_content_type; +3. Delete the contents of `django_content_type`: -Load Database. + ```sql + DELETE FROM hue.django_content_type; + ``` - Add the foreign key, content_type_id, to auth_permission: +4. Load the data, then add the foreign key back: - ALTER TABLE hue.auth_permission ADD FOREIGN KEY (content_type_id) REFERENCES django_content_type (id); + ```sql + ALTER TABLE hue.auth_permission ADD FOREIGN KEY (content_type_id) REFERENCES django_content_type (id); + ``` diff --git a/docs/docs-site/content/administrator/administration/operations.md b/docs/docs-site/content/administrator/administration/operations.md index e1864ef9739..ecb0a20ca62 100644 --- a/docs/docs-site/content/administrator/administration/operations.md +++ b/docs/docs-site/content/administrator/administration/operations.md @@ -5,384 +5,290 @@ draft: false weight: 2 --- -## Admin Wizard +## Hue Administration -The Quick Start wizard allows you to perform the following Hue setup operations by clicking the tab of each step or sequentially by clicking -Next in each screen: +Superusers can open the administration page at `http:///about/`. It contains these tabs: -1. **Check Configuration** validates your Hue configuration. It will note any potential misconfiguration and provide hints as to how to - fix them. You can edit the configuration file described in the next section. -2. **Connectors** The list of services to query or browse -3. **Examples** get started with examples of SQL tables, queries, workflows and jobs to run. -4. **Users** contains a link to the User Admin application to create or import users and a checkbox to enable and disable collection of - usage information. +1. **Overview** has three sections: + * **Config Status** validates your Hue configuration. It lists potential misconfigurations and hints on how to fix them. + * **Examples** installs examples of SQL tables and queries. + * **Analytics** turns the collection of anonymous usage analytics on or off. +2. **Configuration** displays the configuration sections of the installed Hue applications. +3. **Metrics** shows the server metrics. +4. **Server Logs** displays the Hue server log. ### Configuration -Displays a list of the installed Hue applications and their configuration. +Hue ships with a default configuration that assumes a set of services is present. Customize it for your cluster in a `.ini` file: -Hue ships with a default configuration that assumes a various set of services to be present in the cluster. If you are running on a real cluster, you can customize the `hue.ini` configuration file (`/etc/hue/hue.ini` when installed from the package version) or `pseudo-distributed.ini` in `desktop/conf` when in development mode). +* In the installation directory, edit a file in `desktop/conf`, for example `pseudo-distributed.ini` when you build from source. +* If the `HUE_CONF_DIR` environment variable is set, Hue reads the files in that directory instead. -Click the tabs under **Configuration Sections** to see the settings configured for each application. For information on configuring -these settings, see [Configuration](/administrator/configuration/). +Click the tabs under **Configuration Sections** to see the settings of each application. For information on the settings, see [Configuration](/administrator/configuration/). -Hue loads and merges all of the files with extension `.ini` located in the `/etc/hue` directory. Files that are alphabetically later take precedence. +Hue loads and merges all the files with the `.ini` extension in the configuration directory. Files that are alphabetically later take precedence. -After editing the ini file, Hue needs to be restarted. +After editing an `.ini` file, restart Hue. -### Configuration Validation +### Configuration validation -To view the configuration of a running Hue instance, navigate to: +To view the configuration of a running Hue instance, go to: - http:///hue/dump_config +```text +http:///desktop/dump_config +``` -The location of the folder containing the Hue ini files is shown at the top of the page as well as all the configuration values with help and information on the default values. +The page shows the location of the configuration files and all the configuration values, with their help and default values. -To list all available configuration options via the command line, run: +To list all the available configuration options on the command line, run: - /usr/share/hue/build/env/bin/hue config_help | less +```bash +build/env/bin/hue config_help | less +``` -### Server Logs +### Server logs -Displays the Hue Server log and allows you to download them to your local system in a zip file. - -### Threads - -Threads page can be very helpful in debugging purposes. It includes a daemonic thread and the thread objects serving concurrent requests. The host name, thread name identifier and current stack frame of each are displayed. Those are useful when Hue “hangs”, sometimes in case of a request too CPU intensive. There is also a REST API to get the dump of Threads using `desktop/debug/threads`. +The **Server Logs** tab displays the Hue server log. You can download the logs to your computer in a zip file. ### Metrics -Hue uses the **PyFormance** Python library to collect the metrics. These metrics are represented as gauge, counters, meter, rate of events over time, histogram, statistical distribution of values. A REST API endpoint `/desktop/metrics/` to get all the metrics dump as json is also exposed +Hue uses the **PyFormance** Python library to collect metrics. The metrics are gauges, counters, meters (rate of events over time), and histograms (statistical distribution of values). The `/desktop/metrics/` endpoint returns all the metrics as JSON. -The below metrics of most concern to us are displayed on the page: +The **Metrics** tab shows, among others: -- requests.active -- requests.exceptions -- requests.response-time -- threads.daemon -- threads.total -- users -- users.active +* `requests.active` +* `requests.exceptions` +* `requests.response-time` +* `threads.daemon` +* `threads.total` +* `users` +* `users.active` -One of the most useful ones are the percentiles of response time of requests and the count of active users. -Admins can either filter a particular property in all the metrics or select a particular metric for all properties +The most useful ones are the percentiles of the request response time and the count of active users. You can filter a property across all the metrics, or select a metric to see all its properties. ### Logging -The Hue logs are found in `/var/log/hue`, or in a `logs` directory under your -Hue installation root. Inside the log directory you can find: +The Hue logs are in the `logs` directory under your Hue installation root. To use another directory, start the supervisor with the `--log-dir` option. The log directory contains: -* An `access.log` file, which contains a log for all requests against the Hue web server. -* A `supervisor.log` file, which contains log information for the supervisor process. -* A `supervisor.out` file, which contains the stdout and stderr for the supervisor process. -* A `.log` file for each supervised process described above, which contains the logs for that process. -* A `.out` file for each supervised process described above, which contains the stdout and stderr for that process. +* `access.log`: all the requests against the Hue web server. +* `supervisor.log`: the log of the supervisor process. +* `supervisor.out`: the stdout and stderr of the supervisor process. +* A `.log` file for each supervised process: the logs of that process. +* A `.out` file for each supervised process: the stdout and stderr of that process. -If users on your cluster have problems running Hue, you can often find error messages in these log files. If you are unable to start Hue from the init script, the `supervisor.log` log file can often contain clues. +If users have problems running Hue, you can often find error messages in these log files. If you cannot start Hue, the `supervisor.log` file can contain clues. -In addition to logging `INFO` level messages to the `logs` directory, the Hue web server keeps a small buffer of log messages at all levels in memory. You can view these logs by visiting `http://myserver:8888/hue/logs`. The `DEBUG` level messages shown can sometimes be helpful in troubleshooting issues. +Besides logging `INFO` level messages to the `logs` directory, the Hue web server keeps a small buffer of log messages at all levels in memory. To view them, go to `http://myserver:8888/logs`. The `DEBUG` level messages can help you troubleshoot. ## Commands -Commands are utils handy for performing some tasks like update some user password, creating a super user, cleaning-up old sessions and documents. +Commands are utilities for tasks like changing a user password, creating a superuser, and cleaning up old sessions and documents. -Type the following command from the Hue installation root. +Run the commands from the Hue installation root. For example: - cd /usr/lib/hue (or /opt/cloudera/parcels/CDH-XXXXX/share/hue if using parcels and CM) +```bash +cd /usr/share/hue +``` To list all the available commands: - build/env/bin/hue - - > Type 'hue help ' for help on a specific subcommand. - - Available subcommands: - - [auth] - changepassword - createsuperuser - - [axes] - axes_list_attempts - axes_reset - - [beeswax] - beeswax_install_examples - beeswax_install_examples_tests - close_queries - close_sessions - create_table_query_data - - [contenttypes] - remove_stale_contenttypes - - [desktop] - config_dump - config_help - config_override - config_upgrade - convert_documents - create_desktop_app - create_proxy_app - create_test_fs - create_user_directories - desktop_document_cleanup - generate_mdl - is_db_alive - kt_renewer - ldaptest - runcelery - runcherrypyserver - runcpserver - rungunicornserver - runpylint - sync_documents - syncdb - test - version - - [django] - check - compilemessages - createcachetable - dbshell - diffsettings - dumpdata - flush - inspectdb - loaddata - makemessages - makemigrations - migrate - sendtestemail - shell - showmigrations - sqlflush - sqlmigrate - sqlsequencereset - squashmigrations - startapp - startproject - testserver - - [django_extensions] - admin_generator - clean_pyc - clear_cache - compile_pyc - create_app - create_command - create_jobs - create_template_tags - delete_squashed_migrations - describe_form - drop_test_database - dumpscript - export_emails - find_template - generate_secret_key - graph_models - mail_debug - notes - passwd - pipchecker - print_settings - print_user_for_session - reset_db - runjob - runjobs - runprofileserver - runscript - runserver_plus - set_default_site - set_fake_emails - set_fake_passwords - shell_plus - show_template_tags - show_templatetags - show_urls - sqlcreate - sqldiff - sqldsn - sync_s3 - syncdata - unreferenced_files - update_permissions - validate_templates - - [indexer] - indexer_setup - - [notebook] - dbproxy_server - notebook_setup - send_query_stats - - [oozie] - oozie_setup - - [sessions] - clearsessions - - [staticfiles] - collectstatic - findstatic - runserver - - [useradmin] - import_ldap_group - import_ldap_user - sync_ldap_users_and_groups - useradmin_sync_with_unix +```bash +build/env/bin/hue +``` + +To get help on a command: + +```bash +build/env/bin/hue help +``` + +Commonly used commands: + +| Command | Description | +|---|---| +| `changepassword` | Change the password of a user | +| `createsuperuser` | Create a superuser | +| `migrate` | Create or update the database tables | +| `dumpdata`, `loaddata` | Export and import data as JSON | +| `dbshell` | Open the shell of the Hue database | +| `shell` | Open a Python shell with Hue loaded | +| `config_help` | List all the configuration options | +| `desktop_document_cleanup` | Delete old documents | +| `clearsessions` | Delete expired sessions | +| `close_queries`, `close_sessions` | Close old Hive queries and sessions | +| `sync_ldap_users_and_groups`, `import_ldap_user`, `import_ldap_group` | Import and sync users and groups from LDAP | +| `is_db_alive` | Check that the Hue database is reachable | +| `version` | Print the Hue version | ## Troubleshooting ### Instrumentation -To troubleshoot why Hue is slow or consuming high memory, admin can enable instrumentation by setting the `instrumentation` flag to True. +To find out why Hue is slow or uses a lot of memory, set the `instrumentation` flag to `true`: - [desktop] - instrumentation=true +```ini +[desktop] +instrumentation=true +``` -If `django_debug_mode` is enabled, instrumentation is automatically enabled. This flag appends the response time and the total peak memory used since Hue started for every logged request. +If `django_debug_mode` is enabled, instrumentation is enabled automatically. The flag appends the response time and the peak memory used since Hue started to every logged request. Instrumentation enabled: - [17/Apr/2018 15:18:43 -0700] access INFO 127.0.0.1 admin - "POST /jobbrowser/jobs/ HTTP/1.1" `returned in 97ms (mem: 135mb)` +```text +[17/Apr/2018 15:18:43 -0700] access INFO 127.0.0.1 admin - "POST /jobbrowser/jobs/ HTTP/1.1" `returned in 97ms (mem: 135mb)` +``` Instrumentation not enabled: - [23/Apr/2018 10:59:01 -0700] INFO 127.0.0.1 admin - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 88ms +```text +[23/Apr/2018 10:59:01 -0700] INFO 127.0.0.1 admin - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 88ms +``` ### Change or reset a password -Via the Hue commands, to change the password of the currently logged in Unix user: +To change the password of a user, run the `changepassword` command. It uses the currently logged in Unix user when you omit the username: - build/env/bin/hue changepassword +```bash +build/env/bin/hue changepassword [username] +``` -If you don’t remember the admin username, create a new Hue admin (you will then also be able to login and could change the password of another user in Hue): +If you do not remember the admin username, create a new Hue admin. You can then log in and change the password of another user in Hue: - build/env/bin/hue createsuperuser +```bash +build/env/bin/hue createsuperuser +``` -### Make a certain user admin +### Make a user an admin -It is recommended to just do it as an admin via the [Admin UI](https://gethue.com/password-management-in-hue/). +As an admin, use the [Admin UI](https://gethue.com/password-management-in-hue/). -In case this is not possible (e.g. nobody is admin), doing it on the command line is explained in the examples of the [Python API](/developer/api/#python). +If this is not possible (for example when nobody is admin), use the command line as explained in the examples of the [Python API](/developer/api/#python). -At the last recourse, the database user records can be updated via SQL. +As a last resort, update the user records in the database with SQL. -### Exporting Documents +### Export documents Export all documents: - ./build/env/bin/hue dumpdata desktop.Document2 --indent 2 --natural > data.json +```bash +./build/env/bin/hue dumpdata desktop.Document2 --indent 2 --natural > data.json +``` Export specific documents: -20000013 is the id you can see in the URL of the dashboard. +`20000013` is the ID that you can see in the URL of the document. - ./build/env/bin/hue dumpdata desktop.Document2 --indent 2 --pks=20000013 --natural > data.json +```bash +./build/env/bin/hue dumpdata desktop.Document2 --indent 2 --pks=20000013 --natural > data.json +``` -You can specify more than one id: +To export more than one document, list several IDs: - --pks=20000013,20000014,20000015 +```text +--pks=20000013,20000014,20000015 +``` Load the documents: - ./build/env/bin/hue loaddata data.json +```bash +./build/env/bin/hue loaddata data.json +``` ### Large downloads -Download and export options with limited scalability unless using the [Task Server](/administrator/administration/reference/#task-server) is enabled. Restrict the number of rows or bytes transferred using the [following options](/administrator/configuration/apps/#downloads) respectively in your hue.ini. +Downloading and exporting results does not scale well unless the [Task Server](/administrator/administration/reference/#task-server) is enabled. Restrict the number of rows or bytes transferred with the [download options](/administrator/configuration/apps/#downloads) in your `hue.ini`. -### Slow because too documents +### Slow because of too many documents -When the database has too many entries, it will cause performance issue. The config check will help superuser to find this issue. Login as superuser and go to “Hue Administration”, this sample screenshot will be displayed in the quick start wizard when the tables have too many entries. +Too many entries in the database cause performance issues. The configuration check warns superusers about it: log in as a superuser and open **Hue Administration**. ![Check config too many documents](https://cdn.gethue.com/uploads/2019/03/Doc2CountCheck.png) -To clean up Hue database, go to Hue directory and run following clean up command: +To clean up the Hue database, go to the Hue directory and run: - ./build/env/bin/hue desktop_document_cleanup +```bash +./build/env/bin/hue desktop_document_cleanup +``` ### Too many connections -When getting an error similar to `OperationalError: (1040, 'Too many connections')`, this indicates that the Hue database is overloaded and out of connections. Hue only needs 2 but often the database is used by other services that might "hog" them. Increasing max_connections to around 1000 should be sufficient. e.g. for MySQL, connect to it and set below parameter: +When getting an error similar to `OperationalError: (1040, 'Too many connections')`, the Hue database is out of connections. Hue only needs 2 connections, but other services that use the same database can take them all. Increase `max_connections` to around 1000. For example, in MySQL: + +```sql +SET GLOBAL max_connections = 1000; +``` + +### Test a SQL connection from a pod + +1. Check your Hive version in the SQL Editor: - mysql> SET GLOBAL max_connections = 1000; + ```sql + SELECT version() + ``` -### Testing SQL connection from a Pod + Example result: `2.3.2 r857a9fd8ad725a53bd95c1b2d6612f9b1155f44d` -First check your Hive version in the SQL Editor: +2. List the Hue pods: - SELECT version() - > 2.3.2 r857a9fd8ad725a53bd95c1b2d6612f9b1155f44d + ```bash + kubectl get pods + ``` -Then list the Hue pods: + Example output: - kubectl get pods - > NAME READY STATUS RESTARTS AGE - hue-758466dc77-wpcx8 2/2 Running 0 22h - ingress-nginx-controller-5d6fbbddb6-8kd86 1/1 Running 0 23h - postgres-hue-64c9cc8744-dpk47 1/1 Running 1 47d + ```text + NAME READY STATUS RESTARTS AGE + hue-758466dc77-wpcx8 2/2 Running 0 22h + ingress-nginx-controller-5d6fbbddb6-8kd86 1/1 Running 0 23h + postgres-hue-64c9cc8744-dpk47 1/1 Running 1 47d + ``` -Connect to it: +3. Open a shell in the Hue pod: - kubectl exec -it hue-758466dc77-wpcx8 hue -- bash + ```bash + kubectl exec -it hue-758466dc77-wpcx8 -- bash + ``` -Then get the client files of the same Hive version from: +4. Download the Hadoop and Hive client files of the same version as your Hive from [Hadoop](https://archive.apache.org/dist/hadoop/core) and [Hive](https://archive.apache.org/dist/hive), and set the environment: - https://archive.apache.org/dist/hadoop/core - https://archive.apache.org/dist/hive + ```bash + wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.4/hadoop-2.7.4.tar.gz + wget https://archive.apache.org/dist/hive/hive-2.3.2/apache-hive-2.3.2-bin.tar.gz -And install them: + tar -xvzf hadoop-2.7.4.tar.gz + tar -xvzf apache-hive-2.3.2-bin.tar.gz - sudo apt-get install wget - wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.4/hadoop-2.7.4.tar.gz - wget https://archive.apache.org/dist/hive/hive-2.3.2/apache-hive-2.3.2-bin.tar.gz + export HADOOP_HOME=`pwd`/hadoop-2.7.4 + export HIVE_HOME=`pwd`/apache-hive-2.3.2-bin + export JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64 - tar -xvzf hadoop-2.7.4.tar.gz - tar -xvzf apache-hive-2.3.2-bin.tar.gz + export PATH=$PATH:$HIVE_HOME/bin + ``` - export HADOOP_HOME=`pwd`/hadoop-2.7.4 - export HIVE_HOME=`pwd`/apache-hive-2.3.2-bin - export JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64 +5. Connect with Beeline: - PATH=$PATH:$HIVE_HOME/bin + ```bash + beeline -u 'jdbc:hive2://172.21.0.3:10000' + ``` -And now you are ready to connect: + Example output: - beeline -u 'jdbc:hive2://172.21.0.3:10000' - > SLF4J: Class path contains multiple SLF4J bindings. - SLF4J: Found binding in [jar:file:/usr/share/hue/apache-hive-2.3.2-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] - SLF4J: Found binding in [jar:file:/usr/share/hue/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] - Connecting to jdbc:hive2://172.21.0.3:10000 - Connected to: Apache Hive (version 2.3.2) - Driver: Hive JDBC (version 2.3.2) - Transaction isolation: TRANSACTION_REPEATABLE_READ - Beeline version 2.3.2 by Apache Hive - 0: jdbc:hive2://172.21.0.3:10000> SHOW TABLES; - +--------------------+ - | tab_name | - +--------------------+ - | about | - | amandine_test | - | city_cases | - | cricketer | - | cust1 | - | cust2 | - | customer | - | customers | - | student_info | - | ........ | - | web_logs | - | yash_contact_test | - +--------------------+ - 52 rows selected (0.098 seconds) + ```text + Connecting to jdbc:hive2://172.21.0.3:10000 + Connected to: Apache Hive (version 2.3.2) + Driver: Hive JDBC (version 2.3.2) + Transaction isolation: TRANSACTION_REPEATABLE_READ + Beeline version 2.3.2 by Apache Hive + 0: jdbc:hive2://172.21.0.3:10000> SHOW TABLES; + +--------------------+ + | tab_name | + +--------------------+ + | customers | + | web_logs | + +--------------------+ + ``` Read more on the Hive wiki about the [beeline command line](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell). diff --git a/docs/docs-site/content/administrator/administration/reference.md b/docs/docs-site/content/administrator/administration/reference.md index e220d0aeb5f..812df6e75e1 100644 --- a/docs/docs-site/content/administrator/administration/reference.md +++ b/docs/docs-site/content/administrator/administration/reference.md @@ -5,15 +5,15 @@ draft: false weight: 1 --- -A recommended setup consists in: +A recommended setup consists of: -* 2 Hues and 1 Load Balancer -* Databases: MySQL InnoDB, PostgreSQL, Oracle -* Authentication: [LDAP or Username/Password](/administrator/configuration/server/#authentication) +* 2 Hue servers and 1 load balancer +* A database: MySQL InnoDB, PostgreSQL, or Oracle +* Authentication: [LDAP or username/password](/administrator/configuration/server/#authentication) -Typical setups range from 2 to 5 Hue servers, e.g. 3 Hue servers, 300+ unique users, peaks at 125 users/hour with 300 queries +Typical setups range from 2 to 5 Hue servers. For example, 3 Hue servers can serve 300+ unique users, with peaks of 125 users per hour and 300 queries. -In practice ~50 users / Hue peak time is the rule of thumb. This is accounting for the worse case scenarios and it will go much higher with the upcoming [Task Server](#task-server) and Gunicorn integrations. Most of the scale issues are actually related to resource intensive operations like large download of query results or when an RPC call from Hue to a service is slow (e.g. submitting a query hangs when Hive is slow), not by the number of users. +As a rule of thumb, plan for about 50 users per Hue server at peak time. This accounts for worst-case scenarios. Most scale issues come from resource-intensive operations, like large downloads of query results, or from slow calls from Hue to a service (for example, submitting a query hangs when Hive is slow), not from the number of users. The [Task Server](#task-server) is a beta feature that moves such operations out of the API server. ![Reference Architecture](/images/hue_architecture.png) @@ -22,328 +22,361 @@ In practice ~50 users / Hue peak time is the rule of thumb. This is accounting f ### General -* Hue must be behind a load balancer proxying static files. e.g. NGINX is used for the containers, Cloudera Hue ships with HTTPD. -* Adding more Hue instances behind the load balancer will increase performances by 50 concurrent users. -* Database backend should be such as MySql/Postgres/Oracle. Hue does not work on SQLite as it makes concurrent write calls to the database. -* Check the number of documents in the Hue database. If they are too many (more than 100 000), delete the old records: - Stop the Hue service. Log on to the host of your Hue server. Go to Hue directory and run following clean up [command](/administrator/administration/operations/#commands): +* Put Hue behind a load balancer that proxies the static files. For example, NGINX is used for the containers. +* Each additional Hue instance behind the load balancer supports about 50 more concurrent users. +* Use a database like MySQL, PostgreSQL, or Oracle. Hue does not work well on SQLite because it makes concurrent write calls to the database. +* Check the number of documents in the Hue database. If there are too many (more than 100 000), delete the old records. Stop Hue, go to the Hue installation directory, and run the following [command](/administrator/administration/operations/#commands): - cd /opt/cloudera/parcels/CDH/lib/hue # Hue home directory - ./build/env/bin/hue desktop_document_cleanup + ```bash + ./build/env/bin/hue desktop_document_cleanup + ``` +* Python has memory fragmentation issues that show up in Hue. Check the memory usage of Hue periodically. Browsing HDFS directories with many files, downloading a query result, and copying HDFS files are costly operations memory-wise. -* There are some memory fragmentation issues in Python that manifest in Hue. Check the memory usage of Hue periodically. Browsing HDFS dir with many files, downloading a query result, copying a HDFS files are costly operations memory wise. +The **Config Status** section of Hue Administration (`/about/`) warns about detected risks. Make sure it reports no warnings. -The [Config Check](/administrator/administration/reference/) page of Hue (`/hue/about/`) in the administrator section will warn about detected risks. Make sure it is at zero. +Hue caches SQL metadata throughout the application. The list of tables of a database or the columns of a table are fetched once and reused in the autocomplete, the table browser, and the left and right panels. The logs include the total time taken by each request. -Hue comes with caching of SQL metadata throughout all the application, meaning the list of tables or a database or the column description of a table are only fetched once and re-used in the autocomplete, table browser, left and right panels etc.. The profiling of calls adds in the logs with a total time taken by each request automatically logged. +For example: -e.g. - - [24/Jul/2019 14:17:32 +0000] resource DEBUG GET /jobs Got response in 151ms: {"total":0,"offset":1,"len":1,"coordinatorjobs":[]} - [24/Jul/2019 14:17:32 +0000] access INFO 127.0.0.1 romain - "POST /jobbrowser/api/jobs HTTP/1.1" returned in 157ms (mem: 164mb) +```text +[24/Jul/2019 14:17:32 +0000] resource DEBUG GET /jobs Got response in 151ms: {"total":0,"offset":1,"len":1,"coordinatorjobs":[]} +[24/Jul/2019 14:17:32 +0000] access INFO 127.0.0.1 romain - "POST /jobbrowser/api/jobs HTTP/1.1" returned in 157ms (mem: 164mb) +``` ### Query Editor -* Compare performance of the Hive Query Editor in Hue with the exact same query in a beeline shell against the exact same HiveServer2 instance that Hue is pointing to. This will determine if Hue needs to be investigated or HiveServer2 needs to be investigated. -* Check the logging configuration for HiveServer2 by going to Hive service Configuration in Cloudera Manager. Search for HiveServer2 Logging Threshold and make sure that it is not set to DEBUG or TRACE. If it is, drop the logging level to INFO at a minimum. -* Configure individual dedicated HiveServer2 instances for each Hue instance separate from HiveServer2 instances used by other 3rd party tools or clients, or configure Hue to point to multiple HS2 instances behind a Load Balancer. -* Tune the query timeouts for HiveServer2 (in hive-site.xml) and Impala on the hue_safety_valve or hue.ini: Query Life Cycle -Downloading queries past a few thousands rows will lag and increase CPU/memory usage in Hue by a lot. It is for this we are truncating the results until further improvements. +* Compare the performance of the Hive Query Editor in Hue with the same query in a Beeline shell against the same HiveServer2 instance. This shows whether to investigate Hue or HiveServer2. +* Check the logging level of HiveServer2. Make sure it is not set to `DEBUG` or `TRACE`. If it is, set it to `INFO` at a minimum. +* Configure a dedicated HiveServer2 instance for each Hue instance, separate from the HiveServer2 instances used by other tools or clients. Or configure Hue to point to multiple HiveServer2 instances behind a load balancer. +* Tune the query timeouts of HiveServer2 (in `hive-site.xml`) and Impala (in the `hue.ini`). See [Queries life cycle](#queries-life-cycle). + +Downloading more than a few thousand rows lags and increases the CPU and memory usage of Hue a lot. For this reason, the results are truncated. ## Load Balancers Hue is often run with: -* Cherrypy with [NGINX](http://gethue.com/using-nginx-to-speed-up-hue-3-8-0/) (recommended) -* Cherrypy with HTTPD (built-in when using Cloudera Manager) (recommended) +* CherryPy with [NGINX](https://gethue.com/using-nginx-to-speed-up-hue-3-8-0/) (recommended) +* CherryPy with Apache HTTPD * Gunicorn -* [Apache mod Python](http://gethue.com/how-to-run-hue-with-the-apache-server/) ## Queries life cycle -But what happens to the query results? How long are they kept? Why do they disappear sometimes? Why are some Impala queries are still “in flight” even if they are completed? Each query is using some resources in Impala or HiveServer2. When the users submit a lot of queries, they are going to add up and crash the servers if nothing is done. Here are the latest settings that you can tweak: +Each query uses resources in Impala or HiveServer2. When users submit many queries, the resources add up and can crash the servers if nothing is done. These settings control how long queries and their results are kept. ### Impala -Hue tries to close the query when the user navigates away from the result page (as queries are generally fast, it is ok to close them quick). However, if the user never comes back checking the result of the query or never close the page, the query is going to stay. Impala is going to automatically expire the queries idle for than 10 minutes with the query_timeout_s property. - - [impala] - # If > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work - # (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. - query_timeout_s=600 +By default, Hue closes the Impala query when the user leaves the result page (`close_queries=true`). If the user never closes the page, the query stays open. Impala cancels the queries that are idle for longer than `query_timeout_s`. - # If > 0, the session will be timed out (i.e. cancelled) if Impala does not do any work - # (compute or send back results) for that session within QUERY_TIMEOUT_S seconds (default 1 hour). - session_timeout_s=3600 +```ini +[impala] +# If > 0, the query will be timed out (i.e. cancelled) if Impala does not do any work +# (compute or send back results) for that query within QUERY_TIMEOUT_S seconds. +query_timeout_s=300 -Until this version, the only alternative workaround to close all the queries, is to restart Hue (or Impala). +# If > 0, the session will be timed out (i.e. cancelled) if Impala does not do any work +# (compute or send back results) for that session within SESSION_TIMEOUT_S seconds (default 15 minutes). +session_timeout_s=900 +``` -Note: Impala currently only cancels the query but does not close it. It will be improved in a future version with IMPALA-1575. In the meantime specify a -idle_session_timeout=20 in the Impala flags (“Command Line Argument Advanced Configuration Snippet (Safety Valve)”). This setting is also available in the Hue configuration. +> **Note:** Impala only cancels the idle query. It does not close it. To close idle sessions, set `-idle_session_timeout=20` in the Impala flags. Hue also has a `session_timeout_s` setting. ### Hive -Hue never closes the Hive queries by default (as some queries can take hours of processing time). Also if your query volume is low (e.g. < a few hundreds a day) and you restart HiveServer2 every week, you are probably not affected. To get the same behavior as Impala (and close the query when the user leaves the page), switch on in the hue.ini: - - [beeswax] - # Hue will try to close the Hive query when the user leaves the editor page. - # This will free all the query resources in HiveServer2, but also make its results inaccessible. - close_queries=true - -Some close_query and close_session commands were added: +By default, Hue does not close Hive queries, because some queries take hours. If your query volume is low (fewer than a few hundred a day) and you restart HiveServer2 every week, you are probably not affected. To close the query when the user leaves the page, like for Impala, set this in the `hue.ini`: - build/env/bin/hue close_queries --help +```ini +[beeswax] +# Hue will try to close the Hive query when the user leaves the editor page. +# This will free all the query resources in HiveServer2, but also make its results inaccessible. +close_queries=true +``` - Usage: build/env/bin/hue close_queries [options] (default is 7) +The `close_queries` and `close_sessions` commands close old Hive queries and sessions. Both take an optional age in days (default is 7) and an optional `all` keyword: - Closes the non running queries older than 7 days. If is specified, close the ones of any types. +```bash +build/env/bin/hue close_queries [age_in_days] [all] +build/env/bin/hue close_sessions [age_in_days] [all] +``` -To run, be sure to export these environment variables: +* `close_queries` closes the expired, failed and finished queries older than the given age. With `all`, it closes the queries whatever their state. +* `close_sessions` closes the Beeswax sessions older than the given age. With `all`, it closes the sessions of all the applications. - export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk '{print $9}'`" - export HIVE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk '{print $9}'`/hive-conf" +Run `build/env/bin/hue close_queries --help` to see all the options. -Then for example: +Export `HIVE_CONF_DIR` (for example `/etc/hive/conf`) before you run the commands. Export `HUE_CONF_DIR` too if your Hue configuration is not in `desktop/conf`. - ./build/env/bin/hue close_queries 0 - Closing (all=False) HiveServer2 queries older than 0 days... - 1 queries closed. +For example, to close the finished queries and all the sessions older than 0 days: - ./build/env/bin/hue close_sessions 0 hive - Closing (all=False) HiveServer2 sessions older than 0 days... - 1 sessions closed. +```bash +./build/env/bin/hue close_queries 0 +./build/env/bin/hue close_sessions 0 all +``` -You can then add this commands into a crontab and expire the queries older than N days. +```text +Closing (all=False) HiveServer2 queries older than 0 days... +1 queries closed. 0 queries already closed. +Closing (all=True) HiveServer2 sessions older than 0 days... +Found 1 open HiveServer2 sessions to close +1 sessions closed. +0 sessions skipped because already closed. +``` -Like Impala, HiveServer2 can now automatically expires queries. So tweak hive-site.xml with: +Add these commands to a crontab to expire the queries older than N days. - - hive.server2.session.check.interval - 3000 - The check interval for session/operation timeout, which can be disabled by setting to zero or negative value. - +Like Impala, HiveServer2 can expire queries automatically. Set these properties in `hive-site.xml`: - - hive.server2.idle.session.timeout - 3000 - Session will be closed when it's not accessed for this duration, which can be disabled by setting to zero or negative value. - +```xml + + hive.server2.session.check.interval + 3000 + The check interval for session/operation timeout, which can be disabled by setting to zero or negative value. + - - hive.server2.idle.operation.timeout - 0 - Operation will be closed when it's not accessed for this duration of time, which can be disabled by setting to zero value. With positive value, it's checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). With negative value, it's checked for all of the operations regardless of state - + + hive.server2.idle.session.timeout + 3000 + Session will be closed when it's not accessed for this duration, which can be disabled by setting to zero or negative value. + -**Note** + + hive.server2.idle.operation.timeout + 0 + Operation will be closed when it's not accessed for this duration of time, which can be disabled by setting to zero value. With positive value, it's checked for operations in terminal state only (FINISHED, CANCELED, CLOSED, ERROR). With negative value, it's checked for all of the operations regardless of state + +``` -This is the recommended solution for Hive. User wishing to keep some result for longer can issue a CREATE TABLE AS SELECT … or export the results in Hue. +> **Note:** This is the recommended solution for Hive. To keep results for longer, use `CREATE TABLE AS SELECT` or export the results in Hue. ## Impala and Hive HA -How to optimally configure your Analytic Database for High Availability with Hue and other SQL clients. +This section describes how to configure your analytic database for high availability with Hue and other SQL clients. -HiveServer2 and Impala support High Availability through a "load balancer". One caveat is that Hue's underlying Thrift libraries reuse TCP connections in a pool, a single user session may not have the same Impala or Hive TCP connection. If a TCP connection is balanced away from the previously selected  HiveServer2 or Impalad instance, the user session and its queries can be lost and trigger the "Results have expired" or "Invalid session Id" errors. +HiveServer2 and Impala support High Availability through a "load balancer". One caveat is that Hue's underlying Thrift libraries reuse TCP connections in a pool, a single user session may not have the same Impala or Hive TCP connection. If a TCP connection is balanced away from the previously selected HiveServer2 or Impalad instance, the user session and its queries can be lost and trigger the "Results have expired" or "Invalid session Id" errors. -To prevent sessions from being lost, we need configure the load balancer with "source" algorithm to ensure each Hue instance sends all traffic to a single HiveServer2/Impalad instance. Yes, this is not true load balancing, but a configuration for failover High Availability. HiveSever2 or Impala coordinators already distribute the work across the cluster so this is not an issue. +To prevent sessions from being lost, we need configure the load balancer with "source" algorithm to ensure each Hue instance sends all traffic to a single HiveServer2/Impalad instance. Yes, this is not true load balancing, but a configuration for failover High Availability. HiveSever2 or Impala coordinators already distribute the work across the cluster so this is not an issue. -To enable an optimal load distribution that works for everybody, we can create multiple profiles in our load balancer, per port for Hue clients and non-Hue clients like Hive or Impala. We can configure non-Hue clients to distribute loads with "roundrobin" or "leastconn" and configure Hue clients with "source" (source IP Persistence) on dedicated ports, for example, 10015 for Hive beeline commands, 10016 for Hue, 21051 for Hue-Impala interactions while 25003 for Impala shell. +To enable an optimal load distribution that works for everybody, we can create multiple profiles in our load balancer, per port for Hue clients and non-Hue clients like Hive or Impala. We can configure non-Hue clients to distribute loads with "roundrobin" or "leastconn" and configure Hue clients with "source" (source IP Persistence) on dedicated ports, for example, 10015 for Hive beeline commands, 10016 for Hue, 21051 for Hue-Impala interactions while 25003 for Impala shell. -As shown in above diagram, you can configure the HaProxy to have two different ports associated with different load balancing algorithms. Here is a sample configuration (haproxy.cfg) for Hive and Impala HA on a secure cluster. - - #----------------------- - # main frontend which proxys to the backends - #----------------------- - frontend hiveserver2_front - bind *:10015 ssl crt /path/to/cert_key.pem - mode tcp - option tcplog - default_backend hiveserver2 - #----------------------- - # round robin balancing between the various backends - #----------------------- - # This is the setup for HS2. beeline client connect to load_balancer_host:load_balancer_port. - # HAProxy will balance connections among the list of servers listed below. - backend hiveserver2 - balance roundrobin - mode tcp - server hiveserver2_1 host-2.com:10000 ssl ca-file /path/to/truststore.pem check - server hiveserver2_2 host-3.com:10000 ssl ca-file /path/to/truststore.pem check - server hiveserver2_3 host-1.com:10000 ssl ca-file /path/to/truststore.pem check - - # Setup for Hue or other JDBC-enabled applications. - # In particular, Hue requires sticky sessions. - # The application connects to load_balancer_host:10016, and HAProxy balances - # connections to the associated hosts, where Hive listens for JDBC requests on port 10015. - #----------------------- - # main frontend which proxys to the backends - #----------------------- - frontend hivejdbc_front - bind *:10016 ssl crt /path/to/cert_key.pem - mode tcp - option tcplog - stick match src - stick-table type ip size 200k expire 30m - default_backend hivejdbc - - #----------------------- - # source balancing between the various backends - #----------------------- - # HAProxy will balance connections among the list of servers listed below. - backend hivejdbc - balance source - mode tcp - server hiveserver2_1 host-2.com:10000 ssl ca-file /path/to/truststore.pem check - server hiveserver2_2 host-3.com:10000 ssl ca-file /path/to/truststore.pem check - server hiveserver2_3 host-1.com:10000 ssl ca-file /path/to/truststore.pem check - -

- And here is an example for impala HA configuration on a secure cluster. -

- - - # The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver. - # For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000. - #----------------------- - # main frontend which proxys to the backends - #----------------------- - frontend impala_front - bind *:25003 ssl crt /path/to/cert_key.pem - mode tcp - option tcplog - default_backend impala - #----------------------- - # round robin balancing between the various backends - #----------------------- - backend impala - balance leastconn - mode tcp - server impalad1 host-3.com:21000 ssl ca-file /path/to/truststore.pem check - server impalad2 host-2.com:21000 ssl ca-file /path/to/truststore.pem check - server impalad3 host-4.com:21000 ssl ca-file /path/to/truststore.pem check - - # Setup for Hue or other JDBC-enabled applications. - # In particular, Hue requires sticky sessions. - # The application connects to load_balancer_host:21051, and HAProxy balances - # connections to the associated hosts, where Impala listens for JDBC requests on port 21050. - #----------------------- - # main frontend which proxys to the backends - #----------------------- - frontend impalajdbc_front - bind *:21051 ssl crt /path/to/cert_key.pem - mode tcp - option tcplog - stick match src - stick-table type ip size 200k expire 30m - default_backend impalajdbc - #----------------------- - # source balancing between the various backends - #----------------------- - # HAProxy will balance connections among the list of servers listed below. - backend impalajdbc - balance source - mode tcp - server impalad1 host-3.com:21050 ssl ca-file /path/to/truststore.pem check - server impalad2 host-2.com:21050 ssl ca-file /path/to/truststore.pem check - server impalad3 host-4.com:21050 ssl ca-file /path/to/truststore.pem check - -
Note: “check” is required at end of each line to ensure HaProxy can detect any unreachable Impalad/HiveServer2 server, so HA failover can be successful. Without TCP check, you may hit the “TSocket reads 0 byte” error when the Impalad/HiveServer2 server Hue tries to connect is down.
- -

- After editing the /etc/haproxy/haproxy.cfg file, run following commands to restart HaProxy service and check the service restarts successfully. +As shown in above diagram, you can configure the HaProxy to have two different ports associated with different load balancing algorithms. Here is a sample configuration (haproxy.cfg) for Hive and Impala HA on a secure cluster. + +```text +#----------------------- +# main frontend which proxys to the backends +#----------------------- +frontend hiveserver2_front +bind *:10015 ssl crt /path/to/cert_key.pem +mode tcp +option tcplog +default_backend hiveserver2 +#----------------------- +# round robin balancing between the various backends +#----------------------- +# This is the setup for HS2. beeline client connect to load_balancer_host:load_balancer_port. +# HAProxy will balance connections among the list of servers listed below. +backend hiveserver2 +balance roundrobin +mode tcp +server hiveserver2_1 host-2.com:10000 ssl ca-file /path/to/truststore.pem check +server hiveserver2_2 host-3.com:10000 ssl ca-file /path/to/truststore.pem check +server hiveserver2_3 host-1.com:10000 ssl ca-file /path/to/truststore.pem check + +# Setup for Hue or other JDBC-enabled applications. +# In particular, Hue requires sticky sessions. +# The application connects to load_balancer_host:10016, and HAProxy balances +# connections to the associated hosts, where Hive listens for JDBC requests on port 10015. +#----------------------- +# main frontend which proxys to the backends +#----------------------- +frontend hivejdbc_front +bind *:10016 ssl crt /path/to/cert_key.pem +mode tcp +option tcplog +stick match src +stick-table type ip size 200k expire 30m +default_backend hivejdbc + +#----------------------- +# source balancing between the various backends +#----------------------- +# HAProxy will balance connections among the list of servers listed below. +backend hivejdbc +balance source +mode tcp +server hiveserver2_1 host-2.com:10000 ssl ca-file /path/to/truststore.pem check +server hiveserver2_2 host-3.com:10000 ssl ca-file /path/to/truststore.pem check +server hiveserver2_3 host-1.com:10000 ssl ca-file /path/to/truststore.pem check +``` + +And here is an example for Impala HA configuration on a secure cluster. + + +```text +# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver. +# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000. +#----------------------- +# main frontend which proxys to the backends +#----------------------- +frontend impala_front +bind *:25003 ssl crt /path/to/cert_key.pem +mode tcp +option tcplog +default_backend impala +#----------------------- +# round robin balancing between the various backends +#----------------------- +backend impala +balance leastconn +mode tcp +server impalad1 host-3.com:21000 ssl ca-file /path/to/truststore.pem check +server impalad2 host-2.com:21000 ssl ca-file /path/to/truststore.pem check +server impalad3 host-4.com:21000 ssl ca-file /path/to/truststore.pem check + +# Setup for Hue or other JDBC-enabled applications. +# In particular, Hue requires sticky sessions. +# The application connects to load_balancer_host:21051, and HAProxy balances +# connections to the associated hosts, where Impala listens for JDBC requests on port 21050. +#----------------------- +# main frontend which proxys to the backends +#----------------------- +frontend impalajdbc_front +bind *:21051 ssl crt /path/to/cert_key.pem +mode tcp +option tcplog +stick match src +stick-table type ip size 200k expire 30m +default_backend impalajdbc +#----------------------- +# source balancing between the various backends +#----------------------- +# HAProxy will balance connections among the list of servers listed below. +backend impalajdbc +balance source +mode tcp +server impalad1 host-3.com:21050 ssl ca-file /path/to/truststore.pem check +server impalad2 host-2.com:21050 ssl ca-file /path/to/truststore.pem check +server impalad3 host-4.com:21050 ssl ca-file /path/to/truststore.pem check +``` + +> **Note:** “check” is required at end of each line to ensure HaProxy can detect any unreachable Impalad/HiveServer2 server, so HA failover can be successful. Without TCP check, you may hit the “TSocket reads 0 byte” error when the Impalad/HiveServer2 server Hue tries to connect is down. + +After editing the /etc/haproxy/haproxy.cfg file, run following commands to restart HaProxy service and check the service restarts successfully.

- service haproxy restart - service haproxy status +```bash +service haproxy restart +service haproxy status +```

- Also we need add following blocks into hue.ini: -

+ Also add these blocks to the `hue.ini`: - [impala] - server_port=21051 +```ini +[impala] +server_port=21051 - [beeswax] - hive_server_port=10016 +[beeswax] +hive_server_port=10016 +``` ## Monitoring -Performing a `GET /desktop/debug/is_alive` will return a 200 response if running. +A `GET /desktop/debug/is_alive` request returns a 200 response when Hue is running. ## Behind a Proxy -A Web proxy lets you centralize all the access to a certain URL and prettify the address e.g. +A web proxy centralizes the access to a URL and gives Hue a friendlier address. For example: - ec2-54-247-321-151.compute-1.amazonaws.com --> demo.gethue.com +```text +ec2-54-247-321-151.compute-1.amazonaws.com --> demo.gethue.com +``` -Here is one way to do it with [NGINX](http://gethue.com/using-nginx-to-speed-up-hue-3-8-0/) or [Apache](http://gethue.com/i-put-a-proxy-on-hue/). +You can set it up with [NGINX](https://gethue.com/using-nginx-to-speed-up-hue-3-8-0/) or [Apache](https://gethue.com/i-put-a-proxy-on-hue/). ## Task Server -**Beta Feature** +> **Note:** The task server is a beta feature. -The task server is currently a work in progress to outsource all the blocking or resource intensive operations -outside of the API server. Follow [#1526](https://github.com/cloudera/hue/issues/1526) for more information -on when first usable task will be released. +The task server moves blocking or resource-intensive operations out of the API server. Follow [#1526](https://github.com/cloudera/hue/issues/1526) for the progress. -Until then, here is how to try the task server service. +### Prerequisites -Make sure you have Redis installed and running. +Install and run Redis: - sudo apt-get install redis-server -y +```bash +sudo apt-get install redis-server -y +``` -In hue.ini, telling the API server that the Task Server is available: +### Configure the task server - [desktop] - [[task_server]] - enabled=true +In the `hue.ini`, tell the API server that the task server is available. Point the broker to Redis: -Starting the Task server: +```ini +[desktop] +[[task_server_v2]] +enabled=true +broker_url=redis://localhost:6379/0 +``` - ./build/env/bin/celery worker -l info -A desktop +Start the task server: + +```bash +./build/env/bin/celery -A desktop worker -l info +``` ### Tasks #### Query Task -When the task server is enabled, SQL queries are going to be submitted outside of the Hue servers. - -To configure the storage to use to persist those, edit the `result_file_storage` setting: +When the task server is enabled, SQL queries are submitted outside of the Hue servers. - [desktop] - [[task_server]] - result_file_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "/var/lib/hue/query-results"}}' +To configure the storage that persists the results, edit the `result_storage` setting: -#### Email Task +```ini +[desktop] +[[task_server_v2]] +result_storage='{"backend": "django.core.files.storage.FileSystemStorage", "properties": {"location": "/var/lib/hue/query-results"}}' +``` #### Task Scheduler For schedules configured statically in Python: - ./build/env/bin/celery -A desktop beat -l info +```bash +./build/env/bin/celery -A desktop beat -l info +``` For schedules configured dynamically via a table with Django Celery Beat: - [desktop] - [[task_server]] - beat_enabled=true +```ini +[desktop] +[[task_server_v2]] +beat_enabled=true +``` Then: - ./build/env/bin/celery -A desktop beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler +```bash +./build/env/bin/celery -A desktop beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler +``` Note: the first time the tables need to be created with: - ./build/env/bin/hue migrate +```bash +./build/env/bin/hue migrate +``` ### Flower Web UI to monitor tasks: - ./build/env/bin/pip install flower - ./build/env/bin/celery flower --broker=redis://localhost:6379/0 +```bash +./build/env/bin/pip install flower +./build/env/bin/celery flower --broker=redis://localhost:6379/0 +``` Then open-up [http://localhost:5555/tasks](http://localhost:5555/tasks) diff --git a/docs/docs-site/content/administrator/administration/user-management.md b/docs/docs-site/content/administrator/administration/user-management.md index 5287d08a0e7..332348f5996 100644 --- a/docs/docs-site/content/administrator/administration/user-management.md +++ b/docs/docs-site/content/administrator/administration/user-management.md @@ -11,12 +11,13 @@ users and groups individually, or import them from an LDAP directory. Group permissions define the Hue applications visible to group members when they log into Hue and the application features available to them. -Click the **Hue Administration** icon in the top right navigation bar under your username. +To open the User Admin application, click your username in the top right navigation bar, then click **Manage Users**. ## LDAP -[LDAP or PAM pass-through authentication with Hive or Impala and Impersonation -](http://gethue.com/ldap-or-pam-pass-through-authentication-with-hive-or-impala/). +Read how to use [LDAP or PAM pass-through authentication with Hive or Impala and impersonation](https://gethue.com/ldap-or-pam-pass-through-authentication-with-hive-or-impala/). + +To restrict which LDAP groups can log in, see [Limit who can log in](#limit-who-can-log-in). ## Users @@ -125,66 +126,70 @@ groups. It does not import any new users or groups. ### Reset a password -**Programmatically** - -When a Hue administrator loses their password, a more programmatic approach is required to secure the administrator again. Hue comes with a wrapper around the python interpreter called the “shell” command. It loads all the libraries required to work with Hue at a programmatic level. To start the Hue shell, type the following command from the Hue installation root. - -If using CM: +**With a command** - export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk '{print $9}'`" +Go to the Hue installation directory and run: -Then: +```bash +build/env/bin/hue changepassword [username] +``` - cd /usr/lib/hue (or /opt/cloudera/parcels/CDH-XXXXX/share/hue if using parcels and CM) - build/env/bin/hue shell +When you omit the username, the command uses the currently logged in Unix user. -The following is a small script, that can be executed within the Hue shell, to change the password for a user named “example”: +**With the Hue shell** - from django.contrib.auth.models import User - user = User.objects.get(username='example') - user.set_password('some password') - user.save() +If a Hue administrator loses their password, use the Hue `shell` command. It is a wrapper around the Python interpreter that loads the libraries required to work with Hue. From the Hue installation directory, run: -The script can also be invoked in the shell by using input redirection (assuming the script is in a file named script.py): +```bash +build/env/bin/hue shell +``` - build/env/bin/hue shell < script.py +The following script changes the password of a user named `example`. Run it in the Hue shell: -**Via a command** +```python +from django.contrib.auth.models import User +user = User.objects.get(username='example') +user.set_password('some password') +user.save() +``` -Go on the Hue machine, then in the Hue home directory and either type: +You can also run a script file (here `script.py`) with input redirection: -To change the password of the currently logged in Unix user: +```bash +build/env/bin/hue shell < script.py +``` - build/env/bin/hue changepassword +### Reset an admin -### Reset an Admin +**With the Hue shell** -**Programmatically** +To make a user a Hue admin, start the shell: -How to make a certain user a Hue admin +```bash +build/env/bin/hue shell +``` - build/env/bin/hue shell +Then set these properties: -Then set these properties to true: +```python +from django.contrib.auth.models import User - from django.contrib.auth.models import User +a = User.objects.get(username='hdfs') +a.is_staff = True +a.is_superuser = True +a.set_password('my_secret') +a.save() +``` - a = User.objects.get(username='hdfs') - a.is_staff = True - a.is_superuser = True - a.set_password('my_secret') - a.save() +**With a command** -**Via a command** +If you do not remember the admin username, create a new Hue admin. You can then log in and change the password of another user in Hue: -If you don't remember the admin username, create a new Hue admin (you will then also be able to login and could change the password of another user in Hue): - - build/env/bin/hue createsuperuser - -
-Above works with the `AllowFirstUserBackend`, it might be different if another backend is used. -
+```bash +build/env/bin/hue createsuperuser +``` +> **Note:** This works with the `AllowFirstUserDjangoBackend` authentication backend. It can differ with another backend. ## Groups @@ -209,11 +214,15 @@ import groups from an LDAP directory. 3. Click **Add group**. -### Limit users can login +### Limit who can log in + +To allow only the users of some LDAP groups to log in, set `login_groups` in the `hue.ini`: - [desktop] - [[ldap]] - login_groups=ldap_grp1,ldap_grp2,ldap_grp3 +```ini +[desktop] +[[ldap]] +login_groups=ldap_grp1,ldap_grp2,ldap_grp3 +``` ### Importing Groups @@ -239,55 +248,57 @@ Hue and the application features available to them. A company would like to use the following LDAP users and groups in Hue: - John Smith belonging to team A - Helen Taylor belonging to team B +* John Smith belongs to team A. +* Helen Taylor belongs to team B. Assuming the following access requirements: - Team A should be able to use Beeswax, but nothing else. - Team B should only be able to see the Oozie dashboard with readonly permissions. +* Team A can use Beeswax, but nothing else. +* Team B can only see the Oozie dashboard, with read-only permissions. -In Hue 2 the scenarios can be addressed more appropriately. Users can be imported from LDAP by clicking “Add/Sync LDAP user” in Useradmin > Users: +To implement these requirements, import the users from LDAP by clicking **Add/Sync LDAP user** in **Users**. -Similarly, groups can be imported from LDAP by clicking “Add/Sync LDAP group” in Useradmin > Groups. +Similarly, import the groups from LDAP by clicking **Add/Sync LDAP group** in **Groups**. -If a previously imported user’s information was updated recently, the information in Hue will need to be resynchronized. This can be achieved through the LDAP sync feature: +If the information of a previously imported user changed in LDAP, resynchronize it with the LDAP sync feature. -Part A of the example can be addressed by explicitly allowing access Beeswax for Team A. This is managed in the “Groups” tab of the Useradmin app: +For team A, explicitly allow access to Beeswax in the **Groups** tab of the User Admin application. -The Team A group can be edited by clicking on its name, where access privileges for the group are selectable. Here, the “beeswax.access” permission would be selected and the others would be unselected: +Click the name of the team A group to edit its permissions. Select the `beeswax.access` permission and unselect the others. -Part B of the example can be handled by explicitly defining access for Team B. This can be accomplished by following the same steps in part A, except for Team B. Every permission would be unselected except “oozie.dashboard_jobs_access”: +For team B, follow the same steps. Unselect every permission except `oozie.dashboard_jobs_access`. -By explicitly setting the app level permissions, the apps that these users will be able to see will change. For instance, Helen, who is a member of Team B, will only see the Oozie app available: +With explicit application-level permissions, the users see different applications. For example, Helen, who belongs to team B, only sees the Oozie dashboard. ### Programmatically -Set permissions to a group via a Bash shell: +To set the permissions of a group with a Bash script: - #!/bin/env bash +```bash +#!/usr/bin/env bash - set -eu +set -eu - host="localhost" - port="8888" - credentials="/home/hadoop/hue-config/hue_access_admin.txt" # password file - user="admin" - protocol="http" - group_name="spark02-user" - permissions_id1="10" # jobbrowser.access - permissions_id2="18" # spark.access - list_of_users="members=10&members=9&members=4" # number here is an order number for a user in the list of users +host="localhost" +port="8888" +credentials="/home/hadoop/hue-config/hue_access_admin.txt" # password file +user="admin" +protocol="http" +group_name="spark02-user" +permissions_id1="10" # jobbrowser.access +permissions_id2="18" # spark.access +list_of_users="members=10&members=9&members=4" # number here is an order number for a user in the list of users - password=$(cat /home/hadoop/hue-config/hue_access_admin.txt | sed -n 's/admin:(.*)/\1/p') +password=$(sed -n 's/admin:\(.*\)/\1/p' "${credentials}") - curl ${protocol}://${host}:${port}/hue/accounts/login/?fromModal=true -o /dev/null -D - -c cookies.txt -s - x_csrftoken=$(grep csrftoken cookies.txt | cut -f 7) +curl ${protocol}://${host}:${port}/hue/accounts/login/?fromModal=true -o /dev/null -D - -c cookies.txt -s +x_csrftoken=$(grep csrftoken cookies.txt | cut -f 7) - curl -i -X POST ${protocol}://${host}:${port}/hue/accounts/login/?fromModal=true -d "username=${user}&password=${password}" -o /dev/null -D - -c cookies.txt -b cookies.txt -H "X-CSRFToken: ${x_csrftoken}" -s - x_csrftoken=$(grep csrftoken cookies.txt | cut -f 7) +curl -i -X POST ${protocol}://${host}:${port}/hue/accounts/login/?fromModal=true -d "username=${user}&password=${password}" -o /dev/null -D - -c cookies.txt -b cookies.txt -H "X-CSRFToken: ${x_csrftoken}" -s +x_csrftoken=$(grep csrftoken cookies.txt | cut -f 7) - curl -X POST ${protocol}://${host}:${port}/useradmin/groups/edit/${group_name} - -d "csrfmiddlewaretoken=${x_csrftoken}&name=${group_name}&permissions=${permissions_id1}&permissions=${permissions_id2}&${list_of_users}&is_embeddable=true", - -b cookies.txt -H "X-CSRFToken: ${x_csrftoken}" +curl -X POST ${protocol}://${host}:${port}/useradmin/groups/edit/${group_name} \ + -d "csrfmiddlewaretoken=${x_csrftoken}&name=${group_name}&permissions=${permissions_id1}&permissions=${permissions_id2}&${list_of_users}&is_embeddable=true" \ + -b cookies.txt -H "X-CSRFToken: ${x_csrftoken}" +``` diff --git a/docs/docs-site/content/administrator/configuration/_index.md b/docs/docs-site/content/administrator/configuration/_index.md index 65edc7919af..a33cd9f522f 100644 --- a/docs/docs-site/content/administrator/configuration/_index.md +++ b/docs/docs-site/content/administrator/configuration/_index.md @@ -5,15 +5,13 @@ draft: false weight: 3 --- -The file to edit is `hue.ini` (which is based on this [template](https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini)). The actual values used by the API server can be checked on the [Dump Config](/administrator/administration/operations/#configuration) page. +You configure Hue in the `hue.ini` file. It is based on this [template](https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini). To check the values the server actually uses, open the [Dump Config](/administrator/administration/operations/#configuration) page. +The most important sections to configure are: -The most important sections to configure are about the: +* [Server](/administrator/configuration/server/): for example, the relational database where Hue saves users and queries, the login method (such as LDAP), the look and feel, special features, and the apps to disable. +* [Connectors](/administrator/configuration/connectors/): the data warehouse services you want users to query or browse. These are typically the databases users query with SQL and the filesystems they browse. -* [API Server](/administrator/configuration/server/). e.g. pointing to a relational database where Hue saves users and queries, selecting the login method like LDAP, customizing the look & feel, activating special features, disabling some apps... -* [Connectors](/administrator/configuration/connectors/). e.g. pointing to Data Warehouse services you want to make easy to query or browse. Those are typically which databases we want users to query with SQL or filesystems to browse. +> **Note**: Restart the server after you change `hue.ini`. -**Notes** - -* Restarting the server is currently required after for taking ini changes into consideration -* In development mode (i.e. when using `./build/env/bin/hue runserver` and not `runcpserver`), the correct ini file is `desktop/conf/pseudo-distributed.ini` +> **Note**: In development mode (when you start Hue with `./build/env/bin/hue runserver`), the ini file is `desktop/conf/pseudo-distributed.ini`. diff --git a/docs/docs-site/content/administrator/configuration/apps/_index.md b/docs/docs-site/content/administrator/configuration/apps/_index.md index 1990257c761..343a1641515 100644 --- a/docs/docs-site/content/administrator/configuration/apps/_index.md +++ b/docs/docs-site/content/administrator/configuration/apps/_index.md @@ -7,273 +7,292 @@ weight: 3 ## Editor -The goal of the Editor is to open-up data to more users by making self service querying easy and productive. -It has one of the best SQL autocomplete and many [more features](/administrator/configuration/connectors/#databases). +The goal of the Editor is to open up data to more users by making self-service querying easy and productive. +It has one of the best SQL autocompletes and many [more features](/administrator/configuration/connectors/#databases). -It is available in Editor or Notebook. Dialects can be added to the main `[notebook]` section like this: +The Editor is available in the Editor and Notebook apps. You add dialects to the main `[notebook]` section like this: - [notebook] - - [[interpreters]] +```ini +[notebook] + [[interpreters]] - [[[hive]]] - # The name of the snippet. - name=Hive - # The backend connection to use to communicate with the server. - interface=hiveserver2 + [[[hive]]] + # The name of the snippet. + name=Hive + # The backend connection to use to communicate with the server. + interface=hiveserver2 - [[[mysql]]] - name = MySQL - interface=sqlalchemy - options='{"url": "mysql://root:root@localhost:3306/hue"}' + [[[mysql]]] + name=MySQL + interface=sqlalchemy + options='{"url": "mysql://root:root@localhost:3306/hue"}' +``` -**Tip** Do not forget to uncomment the lines by removing the `#` and editing the sections at the correct levels. +> **Tip**: Remember to uncomment the lines by removing the `#`, and to edit the sections at the correct levels. -The editor supports some global settings. +The Editor supports some global settings. ### Downloads -Download and export options with limited scalability by restricting the number of rows or bytes transferred using the following properties: +You can limit the scalability cost of downloads and exports by restricting the number of rows or bytes transferred: - [beeswax] - # A limit to the number of rows that can be downloaded from a query before it is truncated. - # A value of -1 means there will be no limit. - download_row_limit=-1 +```ini +[beeswax] + # A limit to the number of rows that can be downloaded from a query before it is truncated. + # A value of -1 means there is no limit. + download_row_limit=-1 - # A limit to the number of bytes that can be downloaded from a query before it is truncated. - # A value of -1 means there will be no limit. - download_bytes_limit=-1 + # A limit to the number of bytes that can be downloaded from a query before it is truncated. + # A value of -1 means there is no limit. + download_bytes_limit=-1 +``` -In addition, it is possible to disable the download and export feature in the editor, dashboard, as well as in the file browser with the following option in your hue.ini: +You can also disable the download and export feature in the Editor, the Dashboard, and the File Browser: - [desktop] - # Global setting to allow or disable end user downloads in all Hue. - # e.g. Query result in Editors and Dashboards, file in File Browser... - enable_download=false +```ini +[desktop] + # Global setting to allow or disable end user downloads in all Hue. + # e.g. Query result in Editors and Dashboards, file in File Browser... + enable_download=false +``` -The download feature in the file browser can be disabled separately with the following options in your hue.ini: +You can disable the download feature of the File Browser separately: - [filebrowser] - show_download_button=false +```ini +[filebrowser] + show_download_button=false +``` ### Notebook -Enable the Notebook mode which supports multiple snippets of code. +Enable the Notebook mode, which supports multiple snippets of code: - [notebook] - show_notebooks=true +```ini +[notebook] + show_notebooks=true +``` ### External statements -Enable the selection of queries from files, saved queries into the editor or as snippet. +Enable the selection of queries from files or saved queries, into the Editor or as a snippet: - [notebook] - enable_external_statements=false +```ini +[notebook] + enable_external_statements=false +``` ### Batch querying -This option currently only works with Hive and relies on Oozie until [HUE-8738](https://issues.cloudera.org/browse/HUE-8738) gets done. +This option only works with Hive and relies on Oozie. - [notebook] - enable_batch_execute=true +```ini +[notebook] + enable_batch_execute=true +``` -### Assist Query Builder +### Assist query builder -Flag to enable a lightweight SQL query builder where tables and columns can be dragged & dropped from the left table assist. Not to be confused with the [Query Builder](/user/querying/#dashboards). +This flag enables a lightweight SQL query builder. You can drag and drop tables and columns from the left table assist into it. Do not confuse it with the dashboards described in the [Dashboard section](/user/querying/#dashboard). -**Note** This feature is deprecated. +> **Note**: This feature is deprecated. - [notebook] - enable_query_builder=true +```ini +[notebook] + enable_query_builder=true +``` -### Query Analysis +### Query analysis -Display an analysis panel post Impala queries executions with some hints and suggestions. +Show an analysis panel with hints and suggestions after an Impala query runs. -**Note** This feature is experimental. +> **Note**: This feature is experimental. - [notebook] - enable_query_analysis=true +```ini +[notebook] + enable_query_analysis=true +``` -### Query Optimization +### Query optimization -In the `[metadata]` section, Hue is supporting Cloudera Navigator Optimiser and soon other services. The goal is to provide recommendation on how to write better queries and get risk alerts on dangerous operations directly within the [editor](/user/querying/). +In the `[metadata]` section, Hue supports Cloudera Navigator Optimizer, and other services might follow. It gives recommendations on how to write better queries and shows risk alerts on dangerous operations directly in the [Editor](/user/querying/). ### One-click scheduling -Enable the creation of a coordinator for the current SQL query. +Enable the creation of a coordinator for the current SQL query. This feature requires the Oozie app. -**Note** This feature is experimental until Task Server scheduler [HUE-8740](https://issues.cloudera.org/browse/HUE-8740). +> **Note**: This feature is experimental. - [notebook] - enable_query_scheduling=true +```ini +[notebook] + enable_query_scheduling=true +``` ### Credentials -When username or password are not specified in the connection URL, they will be prompted at connection time in the user browser. +When the connection URL does not contain a username or password, Hue prompts the user for them in the browser when connecting. -Parameters are not saved at any time in the Hue database. The are currently not even cached in the Hue process. The clients serves these parameters -each time a query is sent. +Hue does not save these parameters in its database, and it does not cache them in the Hue process. The client sends them each time it submits a query. ## Dashboard -Manually typing SQL is not always the most efficient way to explore a dataset. Dashboards offer visual explorations without typing code. +Typing SQL is not always the most efficient way to explore a dataset. Dashboards offer visual exploration without typing code. -They consist in 3 types: +Hue supports these types: * Interactive multi-widget querying of one source of data -* Query Builder (alpha) * Multi-widget reporting (alpha) -### Solr Search +### Solr search -In the `[search]` section of the configuration file, you should -specify: +In the `[search]` section of the configuration file, specify the URL of the Solr server: - [search] - # URL of the Solr Server - solr_url=http://solr-server.com:8983/solr/ +```ini +[search] + # URL of the Solr server + solr_url=http://solr-server.com:8983/solr/ +``` ### SQL -This application is getting improved via SQL Dashboards and Query Builder [HUE-3228](https://issues.cloudera.org/browse/HUE-3228). - - [dashboard] +The SQL Dashboard is configured in the `[dashboard]` section: - # Activate the Dashboard link in the menu. - ## is_enabled=true +```ini +[dashboard] + # Activate the Dashboard link in the menu. + ## is_enabled=true - # Activate the SQL Dashboard (beta). - ## has_sql_enabled=false + # Activate the SQL Dashboard (beta). + ## has_sql_enabled=false - # Activate the Query Builder (beta). - ## has_query_builder_enabled=false + # Activate the static report layout (beta). + ## has_report_enabled=false - # Activate the static report layout (beta). - ## has_report_enabled=false + # Activate the new grid layout system. + ## use_gridster=true - # Activate the new grid layout system. - ## use_gridster=true + # Activate the widget filter and comparison (beta). + ## has_widget_filter=false - # Activate the widget filter and comparison (beta). - ## has_widget_filter=false + # Activate the tree widget (to drill down fields as dimensions, alpha). + ## has_tree_widget=false - # Activate the tree widget (to drill down fields as dimensions, alpha). - ## has_tree_widget=false + [[engines]] - [[engines]] + # [[[solr]]] + # Requires Solr 6+ + ## analytics=true + ## nesting=false - # [[[solr]]] - # Requires Solr 6+ - ## analytics=true - ## nesting=false - - # [[[sql]]] - ## analytics=true - ## nesting=false + # [[[sql]]] + ## analytics=true + ## nesting=false +``` ## Connectors -Several interfaces are possible and sometimes more than one works for a certain database. When available `HiveServer2` or `SqlAlchemy` are recommended as they are native. +Several interfaces are possible, and sometimes more than one works for a given database. When available, `HiveServer2` or `SqlAlchemy` are recommended because they are native. +### SQLAlchemy -### Sql Alchemy +SQLAlchemy is a robust [connector](https://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine) that natively supports many [SQL dialects](https://docs.sqlalchemy.org/en/latest/dialects). It is the recommended connector for most databases. -SQL Alchemy is a robust [connector](https://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine) that supports -many [SQL dialects](https://docs.sqlalchemy.org/en/latest/dialects) natively. This is the recommended connector for most of the databases. +Install the dialect in the Python system or in the Hue Python virtual environment. For example, for MySQL: -The dialect should be added to the Python system or Hue Python virtual environment. For example for MySQL: +```bash +./build/env/bin/pip install mysqlclient +``` - ./build/env/bin/pip install mysqlclient +Then give Hue the information about the database: -Then give Hue the information about the database source: +```ini +[[[mysql]]] + name=MySQL + interface=sqlalchemy + options='{"url": "mysql://root:root@localhost:3306/hue"}' +``` - [[[mysql]]] - name = MySQL - interface=sqlalchemy - options='{"url": "mysql://root:root@localhost:3306/hue"}' - -**Tip** - -To offer more self service capabilities, parts of the URL can be parameterized and the information will be asked to the user. - -Supported parameters are: - -* USER -* PASSWORD - -e.g. - - mysql://${USER}:${PASSWORD}@localhost:3306/hue +> **Tip**: To offer more self-service, you can parameterize parts of the URL. Hue then asks the user for the values. The supported parameters are `USER` and `PASSWORD`. For example: +> +> ```text +> mysql://${USER}:${PASSWORD}@localhost:3306/hue +> ``` ### HiveServer2 -This is the interface that was created for Apache Hive and Apache Impala. The main advantages are that they provide asynchronous executions and so: +This is the interface created for Apache Hive and Apache Impala. Its main advantages are asynchronous executions, which give you: -* non blocking submission -* progress reports -* non blocking result fetching +* Non-blocking submission +* Progress reports +* Non-blocking result fetching -For example to point to Hive, first configure the `[beeswax]` section. +For example, to point to Hive, first configure the `[beeswax]` section: - [beeswax] - # Host where HiveServer2 is running. - hive_server_host=localhost +```ini +[beeswax] + # Host where HiveServer2 is running. + hive_server_host=localhost - # Port where HiveServer2 Thrift server runs on. - hive_server_port=10000 + # Port where the HiveServer2 Thrift server runs. + hive_server_port=10000 +``` -Then make sure the `hive` interpreter is present in the `[[interpreters]]` list. +Then make sure the `hive` interpreter is in the `[[interpreters]]` list: +```ini +[notebook] [[interpreters]] - [[[hive]]] name=Hive interface=hiveserver2 +``` ### Custom -A series of native connectors (e.g. Livy, ksql...) interacting with the editor have been developed and are listed in the [developer section](/developer/development/#connectors). +Hue provides a series of native connectors (for example Livy and ksqlDB) that interact with the Editor. They are listed in the [developer section](/developer/development/#connectors). ### JDBC -**Note**: This is an historical connector, SQLAlchemy should be prefered at this time as it does not require a proxy and is fully secure. +> **Note**: This is a historical connector. Prefer SQLAlchemy, because it does not require a proxy and is fully secure. + +Use the Editor with any JDBC database. -Use the query editor with any JDBC database. +The `rdbms` interface works well for MySQL, PostgreSQL, SQLite, and Oracle. For other JDBC-compatible databases, Hue supports a `jdbc` interface. -The “rdbms” interface works great for MySQL, PostgreSQL, SQLite, and Oracle, but for other JDBC-compatible databases Hue supports a “jdbc” interface to integrate such databases with the editor. +To integrate an external JDBC database: -Integrating an external JDBC database involves a 3-step process: +1. Download the compatible client driver JAR file for your OS and database. You can usually find the driver files on the database vendor's site. -Download the compatible client driver JAR file for your specific OS and database. Usually you can find the driver files from the official database vendor site. + > **Note**: The MySQL JDBC driver is platform independent, but some drivers are specific to certain operating systems and versions. Check compatibility. -**Note**: In the case of MySQL, the JDBC driver is platform independent, but some drivers are specific to certain OSes and versions so be sure to verify compatibility.) -Add the path to the driver JAR file to your Java CLASSPATH. Here, we set the CLASSPATH environment variable in our `.bash_profile` script. +2. Add the path to the driver JAR file to your Java `CLASSPATH`. For example, set the `CLASSPATH` environment variable in your `.bash_profile`: - # MySQL - export MYSQL_HOME=/Users/hue/Dev/mysql - export CLASSPATH=$MYSQL_HOME/mysql-connector-java-5.1.38-bin.jar:$CLASSPATH + ```bash + # MySQL + export MYSQL_HOME=/Users/hue/Dev/mysql + export CLASSPATH=$MYSQL_HOME/mysql-connector-java-5.1.38-bin.jar:$CLASSPATH + ``` -Add a new interpreter to the notebook app and supply the “name”, set “interface” to jdbc, and set “options” to a JSON object that contains the JDBC connection information. For example, we can connect a local MySQL database named “hue” running on `localhost` and port `8080` via JDBC with the following configuration: +3. Add a new interpreter to the Notebook app. Set `name`, set `interface` to `jdbc`, and set `options` to a JSON object with the JDBC connection information. For example, to connect to a local MySQL database named `hue` that runs on `localhost` and port `3306`: - [notebook] - [[interpreters]] - [[[mysql]]] - name=MySQL JDBC - interface=jdbc - options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": ""}' + ```ini + [notebook] + [[interpreters]] + [[[mysql]]] + name=MySQL JDBC + interface=jdbc + options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": ""}' + ``` -Technically the JDBC is connecting to the database to query via a Java Proxy powered with Py4j. It will automatically -be started if any interpreter is using it. +Under the hood, Hue connects to the database through a Java proxy powered by Py4j. Hue starts the proxy automatically if any interpreter uses it. You can override this behavior: - ## Main flag to override the automatic starting of the DBProxy server. - enable_dbproxy_server=true +```ini +[notebook] + # Main flag to override the automatic starting of the DBProxy server. + enable_dbproxy_server=true +``` -**Note**: -Hue needs to be compiled with the flag `BUILD_DB_PROXY=true` in order to come with the JDBC connector. +> **Note**: Compile Hue with the `BUILD_DB_PROXY=true` flag to get the JDBC connector. -**Tip**: Testing JDBC Configurations -Before adding your interpreter's JDBC configurations to hue.ini, verify that the JDBC driver and connection settings work in a SQL client like SQuirrel SQL. +> **Tip**: Before you add the JDBC configuration to `hue.ini`, check that the JDBC driver and connection settings work in a SQL client such as SQuirreL SQL. -**Tip**: Prompt for JDBC authentication -You can leave out the username and password in the JDBC options, and Hue will instead prompt the user for a username and password. This allows administrators to provide access to JDBC sources without granting all Hue users the same access. +> **Tip**: You can leave the username and password out of the JDBC options. Hue then prompts the user for them. This lets administrators give access to JDBC sources without granting every Hue user the same access. diff --git a/docs/docs-site/content/administrator/configuration/connectors/_index.md b/docs/docs-site/content/administrator/configuration/connectors/_index.md index 50337e38709..fe670b54b71 100644 --- a/docs/docs-site/content/administrator/configuration/connectors/_index.md +++ b/docs/docs-site/content/administrator/configuration/connectors/_index.md @@ -9,120 +9,137 @@ weight: 2 ### Config file -Hue connects to any database or warehouse via native Thrift or SqlAlchemy connectors that need to be added to the [Hue ini file](/administrator/configuration/). Except [impala] and [beeswax] which have a dedicated section, all the other ones should be appended below the [[interpreters]] of [notebook] e.g.: +Hue connects to any database or warehouse via native Thrift, SQLAlchemy or JDBC connectors. Add them to the [hue.ini file](/administrator/configuration/). Except for `[impala]` and `[beeswax]`, which have a dedicated section, add each connector below `[[interpreters]]` in the `[notebook]` section, for example: - [notebook] - [[interpreters]] +```ini +[notebook] +[[interpreters]] - [[[mysql]]] - name=MySQL - interface=sqlalchemy - options='{"url": "mysql://user:password@localhost:3306/hue"}' +[[[mysql]]] +name=MySQL +interface=sqlalchemy +options='{"url": "mysql://user:password@localhost:3306/hue"}' - [[[presto]]] - name = Presto - interface=sqlalchemy - options='{"url": "presto://localhost:8080/hive/default"}' +[[[presto]]] +name = Presto +interface=sqlalchemy +options='{"url": "presto://localhost:8080/hive/default"}' +``` -Note that USER and PASSWORD can be prompted to the user by using variables like `mysql://${USER}:${PASSWORD}@localhost:3306/hue`. +Hue can prompt users for their credentials. Use the `${USER}` and `${PASSWORD}` variables, for example `mysql://${USER}:${PASSWORD}@localhost:3306/hue`. -Most of the interpreters require to install their SqlAlchemy dialect (e.g. `./build/env/bin/pip install pyhive`) either in the global Python environment or in the Hue virtual environment. +Most interpreters require you to install their SQLAlchemy dialect, for example `./build/env/bin/pip install pyhive`. Install it in the global Python environment or in the Hue virtual environment. -Read about [how to build your own parser](/developer/development/#sql-parsers) if you are looking at better autocompletes for your the SQL dialects you use. +To get better autocomplete for your SQL dialect, read [how to build your own parser](/developer/development/#sql-parsers). ### Connectors -Admins can configure the connectors via the UI or [API](/developer/api/rest/#connectors). This feature requires the Hue 5 feature flag set and is quite functional despite not being officially released and on by default. +Admins can configure the connectors via the UI or the [REST API](/developer/api/rest/#connectors). This feature is off by default. Enable it with the `enable_connectors` and `enable_hue_5` flags: - [desktop] - enable_connectors=true - enable_hue_5=true +```ini +[desktop] +enable_connectors=true +enable_hue_5=true +``` -**NOTE:** After enabling the above flags, if a `django.db.utils.OperationalError: (1054, "Unknown column 'useradmin_huepermission.connector_id' in 'field list'")` error comes, then try **changing the DB name** in the hue.ini under `[[database]]` because there is no upgrade path and run the migrate command `./build/env/bin/hue migrate`. +> **Note:** After you enable these flags, you might get the error `django.db.utils.OperationalError: (1054, "Unknown column 'useradmin_huepermission.connector_id' in 'field list'")`. There is no upgrade path for this change. Change the database name under `[[database]]` in hue.ini, then run `./build/env/bin/hue migrate`. -Go to `Administer Server` > `Connectors` > `+ Connector` or directly navigate to the page `http://127.0.0.1:8000/hue/desktop/connectors`. +Go to **Administer Server** > **Connectors** > **+ Connector**, or open `http://127.0.0.1:8000/hue/desktop/connectors`. !["Connectors"](https://cdn.gethue.com/uploads/2020/12/hue-connectors-create.png) -Connectors are also configurable via the public [REST API](/developer/api/rest/#connectors). - ## Databases -Here is the list of dialects: +The following sections list the supported databases and their configuration. ### Apache Hive -The [Dev Onboarding](https://docs.gethue.com/developer/development/#apache-hive) documentation demoes the integration. +The [developer documentation](/developer/development/#apache-hive) shows how to set up a Hive test environment. Support is native via a dedicated section. - [beeswax] - # Host where HiveServer2 is running. - # If Kerberos security is enabled, use fully-qualified domain name (FQDN). - hive_server_host=localhost +```ini +[beeswax] +# Host where HiveServer2 is running. +# If Kerberos security is enabled, use fully-qualified domain name (FQDN). +hive_server_host=localhost - # Port where HiveServer2 Thrift server runs on. - hive_server_port=10000 +# Port where HiveServer2 Thrift server runs on. +hive_server_port=10000 +``` Read more about [LDAP or PAM pass-through authentication](http://gethue.com/ldap-or-pam-pass-through-authentication-with-hive-or-impala/) and [High Availability](../server/). -**Note** For historical reason, the name of the configuration section is `[beeswax]`. +> **Note:** For historical reasons, the configuration section is named `[beeswax]`. **Tez** Requires support for sending multiple queries when using Tez (instead of a maximum of just one at the time). You can turn it on with this setting: - [beeswax] - max_number_of_sessions=3 +```ini +[beeswax] +max_number_of_sessions=3 +``` **LLAP** When the LLAP interpreter is added, there are 2 ways to enable connectivity (direct configuration or service discovery). LLAP is added by enabling the following settings: - [notebook] - [[interpreters]] - [[[llap]]] - name=LLAP - interface=hiveserver2 - - [beeswax] - # Direct Configuration - llap_server_host = localhost - llap_server_port = 10500 - llap_server_thrift_port = 10501 - - # or Service Discovery - ## hive_discovery_llap = true - ## hive_discovery_llap_ha = false - # Shortcuts to finding LLAP znode Key - # Non-HA - hiveserver-interactive-site - hive.server2.zookeeper.namespace ex hive2 = /hive2 - # HA-NonKerberized - _llap ex app name llap0 = /llap0_llap - # HA-Kerberized - _llap-sasl ex app name llap0 = /llap0_llap-sasl - ## hive_discovery_llap_znode = /hiveserver2-hive2 +```ini +[notebook] +[[interpreters]] +[[[llap]]] +name=LLAP +interface=hiveserver2 + +[beeswax] +# Direct Configuration +llap_server_host = localhost +llap_server_port = 10500 +llap_server_thrift_port = 10501 + +# or Service Discovery +## hive_discovery_llap = true +## hive_discovery_llap_ha = false +# Shortcuts to finding LLAP znode Key +# Non-HA - hiveserver-interactive-site - hive.server2.zookeeper.namespace ex hive2 = /hive2 +# HA-NonKerberized - _llap ex app name llap0 = /llap0_llap +# HA-Kerberized - _llap-sasl ex app name llap0 = /llap0_llap-sasl +## hive_discovery_llap_znode = /hiveserver2-hive2 +``` **Service Discovery** -When setup, Hue will query zookeeper to find an enabled hiveserver2 or LLAP endpoint. +When service discovery is enabled, Hue queries ZooKeeper to find an enabled HiveServer2 or LLAP endpoint. - [beeswax] - hive_discovery_llap = true - hive_discovery_hs2 = true +```ini +[beeswax] +hive_discovery_llap = true +hive_discovery_hs2 = true +``` -In order to prevent spamming zookeeper, HiveServer2 is cached for the life of the process and llap is cached based on the following setting: +To avoid flooding ZooKeeper, Hue caches the HiveServer2 endpoint for the life of the process. It caches the LLAP endpoint for the number of seconds set by `cache_timeout`: - [beeswax] - cache_timeout = 60 +```ini +[beeswax] +cache_timeout = 60 +``` ### HPL/SQL -It is an Apache open source procedural extension for SQL for Hive users. This [HPL/SQL Editor](https://gethue.com/blog/2022-02-01-hplsql-support/) post demoes the integration. - [[[hplsql]]] - name=Hplsql - interface=hiveserver2 -**Note:** -- HPL/SQL uses the `[beeswax]` config like Hive uses. -- In hplsql mode, you must terminate the commands using the forward slash character (/). The semicolon (;) is used throughout procedure declarations and can no longer be relied upon to terminate a query in the editor. +HPL/SQL is an Apache open source procedural extension for SQL for Hive users. This [HPL/SQL Editor](https://gethue.com/blog/2022-02-01-hplsql-support/) post demoes the integration. + +```ini +[[[hplsql]]] +name=Hplsql +interface=hiveserver2 +``` + +> **Note:** +> +> - HPL/SQL uses the `[beeswax]` config like Hive uses. +> - In HPL/SQL mode, end commands with a forward slash (`/`). The semicolon (`;`) is used inside procedure declarations, so it can't terminate a query in the editor. ### Apache Impala @@ -130,12 +147,14 @@ This [Impala SQL Editor](https://gethue.com/blog/quickstart-sql-editor-for-apach Support is native via a dedicated section. - [impala] - # Host of the Impala Server (one of the Impalad) - server_host=localhost +```ini +[impala] +# Host of the Impala Server (one of the Impalad) +server_host=localhost - # Port of the Impala Server - server_port=21050 +# Port of the Impala Server +server_port=21050 +``` Read more about [LDAP or PAM pass-through authentication](http://gethue.com/ldap-or-pam-pass-through-authentication-with-hive-or-impala/) and [High Availability](../server/). @@ -144,57 +163,69 @@ Read more about [LDAP or PAM pass-through authentication](http://gethue.com/ldap The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install mysqlclient +```bash +./build/env/bin/pip install mysqlclient +``` Then give Hue the information about the database source: - [[[mysql]]] - name=MySQL - interface=sqlalchemy - options='{"url": "mysql://root:root@localhost:3306/hue"}' - ## mysql://${USER}:${PASSWORD}@localhost:3306/hue +```ini +[[[mysql]]] +name=MySQL +interface=sqlalchemy +options='{"url": "mysql://root:root@localhost:3306/hue"}' +## mysql://${USER}:${PASSWORD}@localhost:3306/hue +``` Query string options are documented in the [SqlAlchemy MySQL documentation](https://docs.sqlalchemy.org/en/latest/dialects/mysql.html). Alternative: - [[[mysqljdbc]]] - name=MySql JDBC - interface=jdbc - ## Specific options for connecting to the server. - ## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable. - ## If 'user' and 'password' are omitted, they will be prompted in the UI. - options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": "root"}' - ## options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver"}' +```ini +[[[mysqljdbc]]] +name=MySql JDBC +interface=jdbc +## Specific options for connecting to the server. +## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable. +## If 'user' and 'password' are omitted, they will be prompted in the UI. +options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver", "user": "root", "password": "root"}' +## options='{"url": "jdbc:mysql://localhost:3306/hue", "driver": "com.mysql.jdbc.Driver"}' +``` ### Presto Presto has been forked into [Trino](#trino) and both share the same configuration. -Currently just substitute 'presto' with 'trino' or vice versa. +To switch between them, replace `presto` with `trino` or vice versa. ### Trino -To support Trino in Hue, we're leveraging the official [Trino Python client](https://github.com/trinodb/trino-python-client) +Hue uses the official [Trino Python client](https://github.com/trinodb/trino-python-client). Hue pins version 0.329.0 of the [`trino` package](https://pypi.org/project/trino/). To install it manually, run: -Install at least the 0.329.0 version of trino from https://github.com/trinodb/trino-python-client or https://pypi.org/project/trino/ +```bash +./build/env/bin/pip install trino +``` - ./build/env/bin/pip install trino +Then give Hue the information about the Trino cluster: -Then give Hue the information about the trino cluster: +```ini +[[[trino]]] +name=Trino +interface=trino +options='{"url": "http://localhost:8080"}' +``` - [[[trino]]] - name=Trino - interface=trino - options='{"url": "http://localhost:8080"}' +To use basic authentication (LDAP or password file), set the credentials. Trino only accepts them over HTTPS, so Hue ignores them and logs a warning for `http://` URLs: -With basic (LDAP or password file) authentication. Trino only accepts it over HTTPS, so the credentials are ignored (and a warning is logged) for `http://` URLs: +```ini +options='{"url": "https://trino.example.com:443", "auth_username": "hue", "auth_password": "secret"}' +``` - options='{"url": "https://trino.example.com:443", "auth_username": "hue", "auth_password": "secret"}' +To avoid storing the password in the configuration, use a password script. Hue uses the script output as the password: -With a password script, which avoids storing the password in the configuration. Its output is used as the password: - - options='{"url": "https://trino.example.com:443", "auth_username": "hue", "auth_password_script": "/path/to/script"}' +```ini +options='{"url": "https://trino.example.com:443", "auth_username": "hue", "auth_password_script": "/path/to/script"}' +``` Other options: @@ -205,63 +236,85 @@ Other options: Databases are listed per catalog and named `catalog.schema`, e.g. `hive.default`. -**Note** -* Only [basic authentication](https://github.com/trinodb/trino-python-client?tab=readme-ov-file#basic-authentication) or unsecured connections are supported. -* The section name must be `trino` for the native client (`interface=trino`). The SQLAlchemy `trino://` URL of the `interface=sqlalchemy` connector is a different integration. +> **Note:** +> +> - Hue supports only [basic authentication](https://github.com/trinodb/trino-python-client?tab=readme-ov-file#basic-authentication) or unsecured connections. +> - The section name must be `trino` for the native client (`interface=trino`). The SQLAlchemy `trino://` URL of the `interface=sqlalchemy` connector is a different integration. #### Custom connectors -API clients can send a snippet of type `custom` that defines its own interface and options, which lets the Hue server connect to any host. This is disabled by default and can be enabled in the `[notebook]` section with `enable_custom_connectors=true` if the users are trusted. Connectors defined in the configuration are not affected. +API clients can send a snippet of type `custom` that defines its own interface and options, which lets the Hue server connect to any host. This is disabled by default. Enable it only if you trust your users: + +```ini +[notebook] +enable_custom_connectors=true +``` + +Connectors defined in the configuration are not affected. ### Oracle The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install cx_Oracle +```bash +./build/env/bin/pip install cx_Oracle +``` Then give Hue the information about the database source: - [[[oracle]]] - name = Oracle - interface=sqlalchemy - options='{"url": "oracle://user:password@localhost"}' +```ini +[[[oracle]]] +name = Oracle +interface=sqlalchemy +options='{"url": "oracle://user:password@localhost"}' +``` ### PostgreSQL The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install psycopg2 - or - ./build/env/bin/pip install psycopg2-binary +```bash +./build/env/bin/pip install psycopg2 + or +./build/env/bin/pip install psycopg2-binary +``` Then give Hue the information about the database source: - [[[postgresql]]] - name = PostgreSql - interface=sqlalchemy - options='{"url": "postgresql+psycopg2://user:password@localhost:31335/database"}' +```ini +[[[postgresql]]] +name = PostgreSql +interface=sqlalchemy +options='{"url": "postgresql+psycopg2://user:password@localhost:31335/database"}' +``` ### AWS Athena The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install PyAthena +```bash +./build/env/bin/pip install PyAthena +``` Then give Hue the information about the database source: - [[[athena]]] - name = AWS Athena - interface=sqlalchemy - options='{"url": "awsathena+rest://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@athena.${REGION}.amazonaws.com:443/${SCHEMA}?s3_staging_dir=${S3_BUCKET_DIRECTORY}&work_group=${WORK_GROUP}&catalog_name=${CATALOG_NAME}"}' +```ini +[[[athena]]] +name = AWS Athena +interface=sqlalchemy +options='{"url": "awsathena+rest://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@athena.${REGION}.amazonaws.com:443/${SCHEMA}?s3_staging_dir=${S3_BUCKET_DIRECTORY}&work_group=${WORK_GROUP}&catalog_name=${CATALOG_NAME}"}' +``` e.g. - options='{"url": "awsathena+rest://XXXXXXXXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@athena.us-west-2.amazonaws.com:443/default?s3_staging_dir=s3://gethue-athena/scratch&work_group=demo_group&catalog_name=AwsDataCatalog"}' +```ini +options='{"url": "awsathena+rest://XXXXXXXXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@athena.us-west-2.amazonaws.com:443/default?s3_staging_dir=s3://gethue-athena/scratch&work_group=demo_group&catalog_name=AwsDataCatalog"}' +``` -Note: Keys and S3 buckets need to be URL quoted but Hue does it automatically for you. +> **Note:** Keys and S3 buckets need to be URL quoted. Hue does it automatically. @@ -269,55 +322,65 @@ Note: Keys and S3 buckets need to be URL quoted but Hue does it automatically fo This [Phoenix SQL Editor](https://gethue.com/sql-querying-apache-hbase-with-apache-phoenix/) post demoes the integration. -The official Phoenix dialect is already shipped in Hue. However if you want to update it yourself: +Hue already ships the official Phoenix dialect. To update it yourself, run: - ./build/env/bin/pip install phoenixdb +```bash +./build/env/bin/pip install phoenixdb +``` Then give Hue the information about the database source: - [[[phoenix]]] - name=HBase Phoenix - interface=sqlalchemy - options='{"url": "phoenix://sql-phoenix.gethue.com:8765/"}' +```ini +[[[phoenix]]] +name=HBase Phoenix +interface=sqlalchemy +options='{"url": "phoenix://sql-phoenix.gethue.com:8765/"}' +``` If using security: - [[[phoenix]]] - name=HBase Phoenix - interface=sqlalchemy - options='{"url": "phoenix://sql-phoenix.gethue.com:8765", "tls": true, "connect_args": {"authentication": "SPNEGO", "verify": false }}' +```ini +[[[phoenix]]] +name=HBase Phoenix +interface=sqlalchemy +options='{"url": "phoenix://sql-phoenix.gethue.com:8765", "tls": true, "connect_args": {"authentication": "SPNEGO", "verify": false }}' +``` Grant the appropriate hbase rights to the 'hue' user, e.g.: - grant 'hue', 'RWXCA' +```bash +grant 'hue', 'RWXCA' +``` With impersonation: - options='{"url": "phoenix://sql-phoenix.gethue.com:8765", "has_impersonation": true}' +```ini +options='{"url": "phoenix://sql-phoenix.gethue.com:8765", "has_impersonation": true}' +``` **Notes** -1. Existing HBase tables need to be mapped to views +1. Existing HBase tables need to be mapped to views. - ``` - 0: jdbc:phoenix:> CREATE VIEW if not exists "analytics_demo_view" ( pk VARCHAR PRIMARY KEY, "hours"."01-Total" VARCHAR ); - Error: ERROR 505 (42000): Table is read only. (state=42000,code=505) - --> - 0: jdbc:phoenix:> CREATE Table if not exists "analytics_demo" ( pk VARCHAR PRIMARY KEY, "hours"."01-Total" VARCHAR ); - ``` + ```sql + CREATE VIEW IF NOT EXISTS "analytics_demo_view" ( pk VARCHAR PRIMARY KEY, "hours"."01-Total" VARCHAR ); + -- Error: ERROR 505 (42000): Table is read only. (state=42000,code=505) + -- Create a table instead: + CREATE TABLE IF NOT EXISTS "analytics_demo" ( pk VARCHAR PRIMARY KEY, "hours"."01-Total" VARCHAR ); + ``` -2. Tables are seeing as uppercase by Phoenix. When getting started, it is simpler to just create the table via Phoenix. +2. Phoenix sees unquoted table names as uppercase. When getting started, create the table via Phoenix. - ``` - Error: ERROR 1012 (42M03): Table undefined. tableName=ANALYTICS_DEMO (state=42M03,code=1012) - --> - 0: jdbc:phoenix:> select * from "analytics_demo" where pk = "domain.0" limit 5; - ``` + ```sql + -- Error: ERROR 1012 (42M03): Table undefined. tableName=ANALYTICS_DEMO (state=42M03,code=1012) + -- Quote the table name: + SELECT * FROM "analytics_demo" WHERE pk = 'domain.0' LIMIT 5; + ``` -3. Phoenix follows Apache Calcite. Feel free to help improve the [SQL autocomplete](https://docs.gethue.com/developer/development/#sql-parsers) support for it. +3. Phoenix follows Apache Calcite. You can help improve the [SQL autocomplete](/developer/development/#sql-parsers) support for it. 4. The UI (and the underlying SQLAlchemy API) cannot distinguish between 'ANY namespace' and 'empty/Default' namespace @@ -328,21 +391,27 @@ This [Druid SQL Editor](https://gethue.com/quick-task-how-to-query-apache-druid- First, make sure that Hue can talk to Druid via the [pydruid SqlAlchemy connector](https://github.com/druid-io/pydruid). - ./build/env/bin/pip install pydruid +```bash +./build/env/bin/pip install pydruid +``` In the hue.ini configuration file, now let's add the interpreter. Here 'druid-host.com' would be the machine where Druid is running. - [notebook] - [[interpreters]] - [[[druid]]] - name = Druid - interface=sqlalchemy - options='{"url": "druid://druid-host.com:8082/druid/v2/sql/?header=true"}' +```ini +[notebook] +[[interpreters]] +[[[druid]]] +name = Druid +interface=sqlalchemy +options='{"url": "druid://druid-host.com:8082/druid/v2/sql/?header=true"}' +``` -`?header=true` option requires Druid 13.0 or later. -Adding the `+https` prefix will use HTTPS e.g.: +The `?header=true` option requires Druid 13.0 or later. +Add the `+https` prefix to use HTTPS, for example: - druid+https://druid-host.com:8082/druid/v2/sql/?header=true +```text +druid+https://druid-host.com:8082/druid/v2/sql/?header=true +``` ### Apache Flink @@ -353,14 +422,16 @@ The dialect currently requires the [Flink SQL Gateway](https://github.com/verver Then add a Flink interpreter in the Hue configuration: - [notebook] +```ini +[notebook] - [[interpreters]] +[[interpreters]] - [[[flink]]] - name=Flink - interface=flink - options='{"url": "http://172.18.0.7:8083"}' +[[[flink]]] +name=Flink +interface=flink +options='{"url": "http://172.18.0.7:8083"}' +``` ### ksqlDB @@ -369,34 +440,40 @@ This [ksqlDB Stream SQL Editor](https://gethue.com/blog/tutorial-query-live-data The ksql Python module should be added to the system or Hue Python virtual environment: - ./build/env/bin/pip install ksql - +```bash +./build/env/bin/pip install ksql +``` -**Note** The connector requires [Hue with Python 3](/administrator/installation/dependencies/#python) Then give Hue the information about the interpreter and ksqlDB server. To add to the list of interpreters: - [[[ksqlDB]]] - name=ksqlDB - interface=ksql - options='{"url": "http://ksqldb-server:8088"}' +```ini +[[[ksqlDB]]] +name=ksqlDB +interface=ksql +options='{"url": "http://ksqldb-server:8088"}' +``` ### Google BigQuery The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install pybigquery - ./build/env/bin/pip install pyasn1==0.4.1 +```bash +./build/env/bin/pip install pybigquery +./build/env/bin/pip install pyasn1==0.4.1 +``` From https://github.com/mxmzdlv/pybigquery. Then give Hue the information about the database source: - [[[bigquery]]] - name = BigQuery - interface=sqlalchemy - options='{"url": "bigquery://project-XXXXXX", "credentials_json": "{\"type\": \"service_account\", ...}"}' +```ini +[[[bigquery]]] +name = BigQuery +interface=sqlalchemy +options='{"url": "bigquery://project-XXXXXX", "credentials_json": "{\"type\": \"service_account\", ...}"}' +``` Where to get the Json credentials? By creating a service account: @@ -410,72 +487,92 @@ Where to get the names? In the 'Resources' panel of Big Query UI: To restrict to one dataset: - options='{"url": "bigquery://project-XXXXXX/dataset_name"}' +```ini +options='{"url": "bigquery://project-XXXXXX/dataset_name"}' +``` Supporting additional [connection parameters](https://github.com/mxmzdlv/pybigquery#connection-string-parameters): - options='{"url": "bigquery://", "use_query_cache": "true"}' +```ini +options='{"url": "bigquery://", "use_query_cache": "true"}' +``` ### Materialize The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install psycopg2 +```bash +./build/env/bin/pip install psycopg2 - or + or - ./build/env/bin/pip install psycopg2-binary +./build/env/bin/pip install psycopg2-binary +``` Then give Hue the information about the database source: - [[[postgresql]]] - name = Materialize - interface=sqlalchemy - options='{"url": "postgresql://user:password@localhost:6875/"}' +```ini +[[[postgresql]]] +name = Materialize +interface=sqlalchemy +options='{"url": "postgresql://user:password@localhost:6875/"}' +``` ### Teradata The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-teradata +```bash +./build/env/bin/pip install sqlalchemy-teradata +``` Then give Hue the information about the database source: - [[[teradata]]] - name = Teradata - interface=sqlalchemy - options='{"url": "teradata://user:password@localhost"}' +```ini +[[[teradata]]] +name = Teradata +interface=sqlalchemy +options='{"url": "teradata://user:password@localhost"}' +``` Alternative: - [[[teradata]]] - name=Teradata JDBC - interface=jdbc - options='{"url": "jdbc:teradata://sqoop-teradata-1400.sjc.cloudera.com/sqoop", "driver": "com.teradata.jdbc.TeraDriver", "user": "sqoop", "password": "sqoop"}' +```ini +[[[teradata]]] +name=Teradata JDBC +interface=jdbc +options='{"url": "jdbc:teradata://localhost/database", "driver": "com.teradata.jdbc.TeraDriver", "user": "user", "password": "password"}' +``` ### DB2 The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install ibm_db_sa +```bash +./build/env/bin/pip install ibm_db_sa +``` (or via https://github.com/ibmdb/python-ibmdbsa/tree/master/ibm_db_sa) Then give Hue the information about the database source: - [[[db2]]] - name = DB2 - interface=sqlalchemy - options='{"url": "db2+ibm_db://user:password@hostname[:port]/database"}' +```ini +[[[db2]]] +name = DB2 +interface=sqlalchemy +options='{"url": "db2+ibm_db://user:password@hostname[:port]/database"}' +``` Alternative: - [[[db2]]] - name=DB2 JDBC - interface=jdbc - options='{"url": "jdbc:db2://db2.vpc.cloudera.com:50000/SQOOP", "driver": "com.ibm.db2.jcc.DB2Driver", "user": "DB2INST1", "password": "cloudera"}' +```ini +[[[db2]]] +name=DB2 JDBC +interface=jdbc +options='{"url": "jdbc:db2://localhost:50000/database", "driver": "com.ibm.db2.jcc.DB2Driver", "user": "user", "password": "password"}' +``` ### Apache Spark SQL @@ -488,44 +585,54 @@ There are two ways to connect depending on your infrastructure: #### Distributed SQL Engine -Hue supports two interfaces: SqlAlchemy and native Thrift. Native Thrift should support better long running queries better, but might have some nits here and there. +Hue supports two interfaces: SqlAlchemy and native Thrift. Native Thrift handles long-running queries better, but it might have some rough edges. ##### SqlAlchemy With SqlAlchemy the dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install git+https://github.com/gethue/PyHive +```bash +./build/env/bin/pip install git+https://github.com/gethue/PyHive +``` -Note: SqlAlchemy interface requires the Hive connector which does not work out of the box because of the issue [#150](https://github.com/dropbox/PyHive/issues/150). But Hue ships and show a slightly patched module that works. +> **Note:** The SQLAlchemy interface requires the Hive connector, which does not work out of the box because of PyHive issue [#150](https://github.com/dropbox/PyHive/issues/150). Install the patched fork shown above. This module might also be needed: - ./build/env/bin/pip install thrift_sasl +```bash +./build/env/bin/pip install thrift_sasl +``` Then give Hue the information about the database source: - [[[sparksql]]] - name=Spark SQL - interface=sqlalchemy - options='{"url": "hive://user:password@localhost:10000/database"}' +```ini +[[[sparksql]]] +name=Spark SQL +interface=sqlalchemy +options='{"url": "hive://user:password@localhost:10000/database"}' +``` ##### Distributed SQL Engine / Thrift Server With the HiveServer Thrift (same as the one used by Hive and Impala so more robust depending on the use cases): - [spark] - # Host of the Spark Thrift Server - # https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html - sql_server_host=localhost +```ini +[spark] +# Host of the Spark Thrift Server +# https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html +sql_server_host=localhost - # Port of the Spark Thrift Server - sql_server_port=10000 +# Port of the Spark Thrift Server +sql_server_port=10000 +``` And make sure you have a `sparksql` interpreter configured: - [[[sparksql]]] - name=Spark SQL - interface=hiveserver2 +```ini +[[[sparksql]]] +name=Spark SQL +interface=hiveserver2 +``` #### Apache Livy @@ -533,33 +640,41 @@ And make sure you have a `sparksql` interpreter configured: This [Spark SQL Editor with Livy](https://gethue.com/blog/quick-task-sql-editor-for-apache-spark-sql-with-livy/) post demoes the integration. - [spark] - # The Livy Server URL. - livy_server_url=http://localhost:8998 +```ini +[spark] +# The Livy Server URL. +livy_server_url=http://localhost:8998 +``` And as always, make sure you have an interpreter configured: - [[[sparksql]]] - name=Spark SQL - interface=livy +```ini +[[[sparksql]]] +name=Spark SQL +interface=livy +``` ### Azure SQL Database The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install pyodbc +```bash +./build/env/bin/pip install pyodbc +``` Then configure ODBC according to the [documentation](https://github.com/mkleehammer/pyodbc). Then give Hue the information about the database source: - [[[azuresql]]] - name = Azure SQL Server - interface=sqlalchemy - options='{"url": "mssql+pyodbc://@:@.database.windows.net:1433/?driver=ODBC+Driver+13+for+SQL+Server"}' +```ini +[[[azuresql]]] +name = Azure SQL Server +interface=sqlalchemy +options='{"url": "mssql+pyodbc://@:@.database.windows.net:1433/?driver=ODBC+Driver+13+for+SQL+Server"}' +``` -Note: Properties need to be URL quoted (e.g. with `urllib.quote_plus(...)` in Python). +> **Note:** Properties need to be URL quoted, for example with `urllib.parse.quote_plus(...)` in Python. Read more on the [Azure SQL Database](https://docs.microsoft.com/en-us/azure/sql-database/sql-database-connect-query-python). @@ -567,59 +682,75 @@ Read more on the [Azure SQL Database](https://docs.microsoft.com/en-us/azure/sql The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install pymssql +```bash +./build/env/bin/pip install pymssql +``` Then give Hue the information about the database source: - [[[mssql]]] - name = SQL Server - interface=sqlalchemy - options='{"url": "mssql+pymssql://:@/?charset=utf8"}' +```ini +[[[mssql]]] +name = SQL Server +interface=sqlalchemy +options='{"url": "mssql+pymssql://:@/?charset=utf8"}' +``` Alternative: Microsoft’s SQL Server JDBC drivers can be downloaded from the official site: [Microsoft JDBC Driver](https://docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server) - [[[sqlserver]]] - name=SQLServer JDBC - interface=jdbc - options='{"url": "jdbc:microsoft:sqlserver://localhost:1433", "driver": "com.microsoft.jdbc.sqlserver.SQLServerDriver", "user": "admin": "password": "pass"}' +```ini +[[[sqlserver]]] +name=SQLServer JDBC +interface=jdbc +options='{"url": "jdbc:microsoft:sqlserver://localhost:1433", "driver": "com.microsoft.jdbc.sqlserver.SQLServerDriver", "user": "admin", "password": "pass"}' +``` ### Vertica The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-vertica-python +```bash +./build/env/bin/pip install sqlalchemy-vertica-python +``` Then give Hue the information about the database source: - [[[vertica]]] - name = Vertica - interface=sqlalchemy - options='{"url": "vertica+vertica_python://user:pwd@localhost:port/database"}' +```ini +[[[vertica]]] +name = Vertica +interface=sqlalchemy +options='{"url": "vertica+vertica_python://user:pwd@localhost:port/database"}' +``` Alternative: Be sure to download the Vertica’s JDBC client driver for the right version and OS. - [[[vertica]]] - name=Vertica JDBC - interface=jdbc - options='{"url": "jdbc:vertica://localhost:5433/example", "driver": "com.vertica.jdbc.Driver", "user": "admin", "password": "pass"}' +```ini +[[[vertica]]] +name=Vertica JDBC +interface=jdbc +options='{"url": "jdbc:vertica://localhost:5433/example", "driver": "com.vertica.jdbc.Driver", "user": "admin", "password": "pass"}' +``` ### AWS Redshift The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-redshift +```bash +./build/env/bin/pip install sqlalchemy-redshift +``` Then give Hue the information about the database source: - [[[redshift]]] - name = Redshift - interface=sqlalchemy - options='{"url": "redshift+psycopg2://username@host.amazonaws.com:5439/database"}' +```ini +[[[redshift]]] +name = Redshift +interface=sqlalchemy +options='{"url": "redshift+psycopg2://username@host.amazonaws.com:5439/database"}' +``` ### Apache Drill @@ -628,55 +759,67 @@ The dialect is available on https://github.com/JohnOmernik/sqlalchemy-drill Then give Hue the information about the database source: - [[[drill]]] - name = Drill - interface=sqlalchemy - options='{"url": "drill+sadrill://..."}' - ## To use Drill with SQLAlchemy you will need to craft a connection string in the format below: - # drill+sadrill://:@:/?use_ssl=True - ## To connect to Drill running on a local machine running in embedded mode you can use the following connection string. - # drill+sadrill://localhost:8047/dfs?use_ssl=False +```ini +[[[drill]]] +name = Drill +interface=sqlalchemy +options='{"url": "drill+sadrill://..."}' +## To use Drill with SQLAlchemy you will need to craft a connection string in the format below: +# drill+sadrill://:@:/?use_ssl=True +## To connect to Drill running on a local machine running in embedded mode you can use the following connection string. +# drill+sadrill://localhost:8047/dfs?use_ssl=False +``` Alternative: The Drill JDBC driver can be used. - [[[drill]]] - name=Drill JDBC - interface=jdbc - ## Specific options for connecting to the server. - ## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable. - ## If 'user' and 'password' are omitted, they will be prompted in the UI. - options='{"url": "", "driver": "org.apache.drill.jdbc.Driver", "user": "admin", "password": "admin"}' +```ini +[[[drill]]] + name=Drill JDBC + interface=jdbc + ## Specific options for connecting to the server. + ## The JDBC connectors, e.g. mysql.jar, need to be in the CLASSPATH environment variable. + ## If 'user' and 'password' are omitted, they will be prompted in the UI. + options='{"url": "", "driver": "org.apache.drill.jdbc.Driver", "user": "admin", "password": "admin"}' +``` ### SAP Sybase The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install python-sybase +```bash +./build/env/bin/pip install python-sybase +``` Then give Hue the information about the database source: - [[[sybase]]] - name = Sybase - interface=sqlalchemy - options='{"url": "sybase+pysybase://:@/[database name]"}' +```ini +[[[sybase]]] +name = Sybase +interface=sqlalchemy +options='{"url": "sybase+pysybase://:@/[database name]"}' +``` ### SAP Hana The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-hana +```bash +./build/env/bin/pip install sqlalchemy-hana +``` (or via https://github.com/SAP/sqlalchemy-hana) Then give Hue the information about the database source: - [[[db2]]] - name = DB2 - interface=sqlalchemy - options='{"url": "hana://username:password@example.de:30015"}' +```ini +[[[hana]]] +name = SAP HANA +interface=sqlalchemy +options='{"url": "hana://username:password@example.de:30015"}' +``` ### Apache Solr @@ -686,44 +829,52 @@ Query Solr collections like you would query a regular database via [Solr SQL](ht The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-solr +```bash +./build/env/bin/pip install sqlalchemy-solr +``` (or via https://github.com/aadel/sqlalchemy-solr) Then give Hue the information about the database source: - [[[solr]]] - name = Solr SQL - interface=sqlalchemy - options='{"url": "solr://:@:/solr/[?use_ssl=true|false]"}' +```ini +[[[solr]]] +name = Solr SQL +interface=sqlalchemy +options='{"url": "solr://:@:/solr/[?use_ssl=true|false]"}' +``` **Note** -There is also a native implementation which has some caveats ([HUE-3686](https://issues.cloudera.org/browse/HUE-3686) but reuses the `Dashboard` configuration which is builtin in CDH/CDP. +There is also a native implementation, which has some caveats. It reuses the `[search]` configuration of the [Dashboards](#dashboards). -First make sure Solr is configured for Dashboards (cf. section just below): +First make sure Solr is configured for Dashboards, as described below. Then add the interpreter: - [[[solr]]] - name = Solr SQL - interface=solr - ## Name of the collection handler - # options='{"collection": "default"}' +```ini +[[[solr]]] +name = Solr SQL +interface=solr +## Name of the collection handler +# options='{"collection": "default"}' +``` #### Dashboards -Hue ships the [dynamic dashboards](/user/querying/#dashboard)for exploring datasets visually. Just point to an existing Solr server: +Hue ships [dynamic dashboards](/user/querying/#dashboard) for exploring datasets visually. Just point to an existing Solr server: - [search] - # URL of the Solr Server - solr_url=http://localhost:8983/solr/ +```ini +[search] +# URL of the Solr Server +solr_url=http://localhost:8983/solr/ - # Requires FQDN in solr_url if enabled - ## security_enabled=false +# Requires FQDN in solr_url if enabled +## security_enabled=false - ## Query sent when no term is entered - ## empty_query=*:* +## Query sent when no term is entered +## empty_query=*:* +``` ### Apache Kylin @@ -734,68 +885,86 @@ See how to configure the [Kylin Query Editor](http://gethue.com/using-hue-to-int The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install kylinpy +```bash +./build/env/bin/pip install kylinpy +``` Then give Hue the information about the database source: - [[[kylin]]] - name = Kylin - interface=sqlalchemy - options='{"url": "kylin://..."}' +```ini +[[[kylin]]] +name = Kylin +interface=sqlalchemy +options='{"url": "kylin://..."}' +``` Alternative: - [[[kylin]]] - name=kylin JDBC - interface=jdbc - options='{"url": "jdbc:kylin://172.17.0.2:7070/learn_kylin", "driver": "org.apache.kylin.jdbc.Driver", "user": "ADMIN", "password": "KYLIN"}' +```ini +[[[kylin]]] +name=kylin JDBC +interface=jdbc +options='{"url": "jdbc:kylin://172.17.0.2:7070/learn_kylin", "driver": "org.apache.kylin.jdbc.Driver", "user": "ADMIN", "password": "KYLIN"}' +``` ### Dask SQL dask-sql adds a SQL layer on top of Dask. It uses the Presto wire protocol for communication, so the SqlAlchemy dialect for Presto should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install pyhive +```bash +./build/env/bin/pip install pyhive +``` Then give Hue the information about the database source: - [[[dasksql]]] - name=Dask SQL - interface=sqlalchemy - options='{"url": "presto://localhost:8080/catalog/default"}' +```ini +[[[dasksql]]] +name=Dask SQL +interface=sqlalchemy +options='{"url": "presto://localhost:8080/catalog/default"}' +``` ### Clickhouse -The [qlalchemy-clickhouse](https://github.com/xzkostyan/clickhouse-sqlalchemy) dialect should be added to the Python system or Hue Python virtual environment: +The [sqlalchemy-clickhouse](https://pypi.org/project/sqlalchemy-clickhouse/) dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install sqlalchemy-clickhouse +```bash +./build/env/bin/pip install sqlalchemy-clickhouse +``` Then give Hue the information about the database source: - [[[clickhouse]]] - name = Clickhouse - interface=sqlalchemy - # e.g. clickhouse://user:password@example.com:8124/test?protocol=https - options='{"url": "clickhouse://localhost:8123"}' - -Note: +```ini +[[[clickhouse]]] +name = Clickhouse +interface=sqlalchemy +# e.g. clickhouse://user:password@example.com:8124/test?protocol=https +options='{"url": "clickhouse://localhost:8123"}' +``` -If facing an error like: +If you get an error like: - ModuleNotFoundError: No module named 'connector' +```text +ModuleNotFoundError: No module named 'connector' +``` -Update: +Update `pytz`: - ./build/env/bin/pip install pytz==2021.1 +```bash +./build/env/bin/pip install pytz==2021.1 +``` Alternative: - [[[clickhouse]]] - name=ClickHouse - interface=jdbc - ## Specific options for connecting to the ClickHouse server. - ## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable. - options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}' +```ini +[[[clickhouse]]] +name=ClickHouse +interface=jdbc +## Specific options for connecting to the ClickHouse server. +## The JDBC driver clickhouse-jdbc.jar and its related jars need to be in the CLASSPATH environment variable. +options='{"url": "jdbc:clickhouse://localhost:8123", "driver": "ru.yandex.clickhouse.ClickHouseDriver", "user": "readonly", "password": ""}' +``` ### Apache Kyuubi @@ -804,63 +973,81 @@ Alternative: Give Hue the information about the database source: - [[[kyuubi]]] - name=Kyuubi - interface=jdbc - ## Specific options for connecting to the Kyuubi server. - ## The JDBC driver kyuubi-hive-jdbc-shaded.jar and its related jars need to be in the CLASSPATH environment variable. - options='{"url": "jdbc:hive2://localhost:10009", "driver": "org.apache.kyuubi.jdbc.KyuubiHiveDriver", "user": "readonly", "password": ""}' +```ini +[[[kyuubi]]] +name=Kyuubi +interface=jdbc +## Specific options for connecting to the Kyuubi server. +## The JDBC driver kyuubi-hive-jdbc-shaded.jar and its related jars need to be in the CLASSPATH environment variable. +options='{"url": "jdbc:hive2://localhost:10009", "driver": "org.apache.kyuubi.jdbc.KyuubiHiveDriver", "user": "readonly", "password": ""}' +``` ### Elastic Search The dialect for https://github.com/elastic/elasticsearch should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install elasticsearch-dbapi +```bash +./build/env/bin/pip install elasticsearch-dbapi +``` Then give Hue the information about the database source: - [[[es]]] - name = Elastic Search - interface=sqlalchemy - options='{"url": "elasticsearch+http://localhost:9200/"}' +```ini +[[[es]]] +name = Elastic Search +interface=sqlalchemy +options='{"url": "elasticsearch+http://localhost:9200/"}' +``` ### Apache Pinot DB The dialect for https://pinot.apache.org should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install pinotdb +```bash +./build/env/bin/pip install pinotdb +``` Then give Hue the information about the database source: - [[[pinot]]] - name = Pinot - interface=sqlalchemy - options='{"url": "pinot+http://localhost:8099/query?server=http://localhost:9000/"}' +```ini +[[[pinot]]] +name = Pinot +interface=sqlalchemy +options='{"url": "pinot+http://localhost:8099/query?server=http://localhost:9000/"}' +``` ### Snowflake The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install snowflake-sqlalchemy +```bash +./build/env/bin/pip install snowflake-sqlalchemy +``` Then give Hue the information about the database source: - [[[snowflake]]] - name = Snowflake - interface=sqlalchemy - options='{"url": "snowflake://{user}:{password}@{account}/{database}"}' +```ini +[[[snowflake]]] +name = Snowflake +interface=sqlalchemy +options='{"url": "snowflake://{user}:{password}@{account}/{database}"}' +``` Note: account is the name in your URL domain. e.g. - https://smXXXXXX.snowflakecomputing.com/ --> smXXXXXX +```text +https://smXXXXXX.snowflakecomputing.com/ --> smXXXXXX +``` Tables currently need to be prefixed with a schema, e.g. `SELECT * FROM tpch_sf1.customer LIMIT 5` e.g. - options='{"url": "snowflake://hue:pwd@smXXXXX/SNOWFLAKE_SAMPLE_DATA"}' +```ini +options='{"url": "snowflake://hue:pwd@smXXXXX/SNOWFLAKE_SAMPLE_DATA"}' +``` Read more about is on the [snowflake-sqlalchemy page](https://docs.snowflake.net/manuals/user-guide/sqlalchemy.html). @@ -868,23 +1055,29 @@ Read more about is on the [snowflake-sqlalchemy page](https://docs.snowflake.net Just give Hue the information about the database source: - [[[sqlite]]] - name = Sqlite - interface=sqlalchemy - options='{"url": "sqlite:///path/to/database.db"}' +```ini +[[[sqlite]]] +name = Sqlite +interface=sqlalchemy +options='{"url": "sqlite:///path/to/database.db"}' +``` ### Google Sheets The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install gsheetsdb +```bash +./build/env/bin/pip install gsheetsdb +``` Then give Hue the information about the database source: - [[[GSheets]]] - name = Google Sheets - interface=sqlalchemy - options='{"url": "gsheets://"}' +```ini +[[[gsheets]]] +name = Google Sheets +interface=sqlalchemy +options='{"url": "gsheets://"}' +``` Read more on the [gsheetsdb page](https://github.com/betodealmeida/gsheets-db-api#authentication). @@ -892,14 +1085,18 @@ Read more on the [gsheetsdb page](https://github.com/betodealmeida/gsheets-db-ap The dialect should be added to the Python system or Hue Python virtual environment: - ./build/env/bin/pip install psycopg2 +```bash +./build/env/bin/pip install psycopg2 +``` Then give Hue the information about the database source: - [[[greenplum]]] - name = Greenplum - interface=sqlalchemy - options='{"url": "postgresql+psycopg2://user:password@localhost:31335/database"}' +```ini +[[[greenplum]]] +name = Greenplum +interface=sqlalchemy +options='{"url": "postgresql+psycopg2://user:password@localhost:31335/database"}' +``` ## Storage @@ -908,251 +1105,288 @@ Then give Hue the information about the database source: Hue supports one HDFS cluster. That cluster should be defined under the `[[[default]]]` sub-section. - [hadoop] +```ini +[hadoop] - # Configuration for HDFS NameNode - # ------------------------------------------------------------------------ - [[hdfs_clusters]] +# Configuration for HDFS NameNode +# ------------------------------------------------------------------------ +[[hdfs_clusters]] - [[[default]]] - fs_defaultfs=hdfs://hdfs-name-node.com:8020 - webhdfs_url=http://hdfs-name-node.com:20101/webhdfs/v1 +[[[default]]] +fs_defaultfs=hdfs://hdfs-name-node.com:8020 +webhdfs_url=http://hdfs-name-node.com:20101/webhdfs/v1 +``` HA is supported by pointing to the HttpFs service instead of the NameNode. Make sure the HDFS service has in it `hdfs-site.xml`: - - dfs.webhdfs.enable - true - +```xml + + dfs.webhdfs.enable + true + +``` Configure Hue as a proxy user for all other users and groups, meaning it may submit a request on behalf of any other user: WebHDFS: Add to core-site.xml: - - - hadoop.proxyuser.hue.hosts - * - - - hadoop.proxyuser.hue.groups - * - +```xml + + +hadoop.proxyuser.hue.hosts +* + + +hadoop.proxyuser.hue.groups +* + +``` HttpFS: Verify that /etc/hadoop-httpfs/conf/httpfs-site.xml has the following configuration: - - - httpfs.proxyuser.hue.hosts - * - - - httpfs.proxyuser.hue.groups - * - +```xml + + +httpfs.proxyuser.hue.hosts +* + + +httpfs.proxyuser.hue.groups +* + +``` If the configuration is not present, add it to /etc/hadoop-httpfs/conf/httpfs-site.xml and restart the HttpFS daemon. Verify that core-site.xml has the following configuration: - - hadoop.proxyuser.httpfs.hosts - * - - - hadoop.proxyuser.httpfs.groups - * - +```xml + +hadoop.proxyuser.httpfs.hosts +* + + +hadoop.proxyuser.httpfs.groups +* + +``` If the configuration is not present, add it to /etc/hadoop/conf/core-site.xml and restart Hadoop. ### Apache Ozone -Hue can read and write files on the Ozone filesystem, similar to HDFS. Add the following settings under `[desktop]`, and within the `[[ozone]]` section: +Hue can read and write files on the Ozone filesystem, similar to HDFS. Add the following settings in the `[[ozone]]` section under `[desktop]`: + +```ini +[desktop] +[[ozone]] +[[[default]]] +fs_defaultfs=ofs://[***SERVICE-ID***] +webhdfs_url=http://[***OZONE-HTTPFS-HOST***]:[***OZONE-HTTPFS-PORT***]/webhdfs/v1 +ssl_cert_ca_verify=true +security_enabled=true +``` + +Where: + - [[[default]]] - fs_defaultfs=ofs://[***SERVICE-ID***] - webhdfs_url=http://[***OZONE-HTTPFS-HOST***]:[***OZONE-HTTPFS-PORT***]/webhdfs/v1 - ssl_cert_ca_verify=true - security_enabled=true - -Where, - **fs_defaultfs:** Ozone service ID (HA mode) or URL for Ozone Manager (non-HA mode). - **webhdfs_url:** URL of HttpFS endpoint for the running Ozone service (`https` for secure service else `http`). -Configure Hue as a proxy user for all other users and groups, meaning it may submit a request on behalf of any other user: +Configure Hue as a proxy user for all other users and groups, so that it can submit requests on behalf of any other user. -HttpFS: Verify or add the following configuration in Ozone service's `httpfs-site.xml`: +In the Ozone service, verify or add the following configuration in `httpfs-site.xml`: - - - httpfs.proxyuser.hue.hosts - * - - - httpfs.proxyuser.hue.groups - * - +```xml + + +httpfs.proxyuser.hue.hosts +* + + +httpfs.proxyuser.hue.groups +* + +``` ### S3 -Hue's filebrowser can now allow users to explore, manage, and upload data in an S3 account, in addition to HDFS and Ozone. - -Read more about it in the [S3 User Documentation](/user/browsing#s3). +The File Browser lets users explore, manage and upload data in an S3 account, in addition to HDFS and Ozone. -In order to add an S3 account to Hue, you'll need to configure Hue with valid S3 credentials, including the access key ID and secret access key. +Read more in the [S3 user documentation](/user/browsing/#aws-s3). -These keys can securely stored in a script that outputs the actual access key and secret key to stdout to be read by Hue (this is similar to how Hue reads password scripts). In order to use script files, add the following section to your hue.ini configuration file: +To add an S3 account, configure Hue with valid S3 credentials: the access key ID and the secret access key. +You can store the keys securely in scripts that print the key to stdout, in the same way as Hue reads password scripts. To use scripts, add the following section to hue.ini: - [aws] - [[aws_accounts]] - [[[default]]] - access_key_id_script=/path/to/access_key_script - secret_access_key_script= /path/to/secret_key_script - allow_environment_credentials=false - region=us-east-1 +```ini +[aws] +[[aws_accounts]] +[[[default]]] +access_key_id_script=/path/to/access_key_script +secret_access_key_script=/path/to/secret_key_script +allow_environment_credentials=false +region=us-east-1 +``` -Alternatively (but not recommended for production or secure environments), you can set the access_key_id and secret_access_key values to the plain-text values of your keys: +Alternatively, set the plain-text values of your keys. Don't do this in production or secure environments: - [aws] - [[aws_accounts]] - [[[default]]] - access_key_id=s3accesskeyid - secret_access_key=s3secretaccesskey - allow_environment_credentials=false - region=us-east-1 +```ini +[aws] +[[aws_accounts]] +[[[default]]] +access_key_id=s3accesskeyid +secret_access_key=s3secretaccesskey +allow_environment_credentials=false +region=us-east-1 +``` -The region should be set to the AWS region corresponding to the S3 account. By default, this region will be set to 'us-east-1'. +Set `region` to the AWS region of the S3 account. When it is not set, Hue falls back to `us-east-1` (or to the region of the EC2 instance). `region` is ignored when `host` is set. **Using Ceph** -New end points have been added in [HUE-5420](https://issues.cloudera.org/browse/HUE-5420) + +To use an S3-compatible service such as Ceph, set `host` to its endpoint. You can also set `is_secure` and `calling_format`. ### Azure File Systems -Hue's file browser can now allow users to explore, manage, and upload data in an ADLS v1 or ADLS v2 (ABFS), in addition to HDFS, Ozone and S3. +The File Browser lets users explore, manage and upload data in ADLS v1 or ADLS v2 (ABFS), in addition to HDFS, Ozone and S3. -In order to add an Azure account to Hue, you'll need to configure Hue with valid Azure credentials, including the client ID, client secret and tenant ID. -These keys can securely stored in a script that outputs the actual access key and secret key to stdout to be read by Hue (this is similar to how Hue reads password scripts). In order to use script files, add the following section to your hue.ini configuration file: +To add an Azure account, configure Hue with valid Azure credentials: the client ID, client secret and tenant ID. Add them to hue.ini: - [azure] - [[azure_account]] - [[[default]]] - client_id=adlsclientid - client_secret=adlsclientsecret - tenant_id=adlstenantid +```ini +[azure] +[[azure_accounts]] +[[[default]]] +client_id=adlsclientid +client_secret=adlsclientsecret +tenant_id=adlstenantid +``` -The account name used by ADLS / ABFS will need to be configured via the following properties: +Then set the account name used by ADLS or ABFS in the same `[azure]` section: - [[adls_clusters]] - [[[default]]] - fs_defaultfs=adl://.azuredatalakestore.net - webhdfs_url=https://.azuredatalakestore.net +```ini +[azure] +[[adls_clusters]] +[[[default]]] +fs_defaultfs=adl://.azuredatalakestore.net +webhdfs_url=https://.azuredatalakestore.net/webhdfs/v1 - [[abfs_clusters]] - [[[default]]] - fs_defaultfs=abfs://@.dfs.core.windows.net - webhdfs_url=https://@.dfs.core.windows.net +[[abfs_clusters]] +[[[default]]] +fs_defaultfs=abfs://@.dfs.core.windows.net +webhdfs_url=https://.dfs.core.windows.net +``` ### GCS -Hue's file browser for Google Cloud Storage is currently a work in progress with [HUE-8978](https://issues.cloudera.org/browse/HUE-8978) - -The json credentials of a service account can be stored for development in plain-text +The File Browser can explore and manage Google Cloud Storage buckets (`gs://`) with a service account. For development, you can store its JSON credentials in plain text: - [desktop] - [[gc_accounts]] - [[[default]]] - json_credentials='{ "type": "service_account", "project_id": .... }' +```ini +[desktop] +[[gc_accounts]] +[[[default]]] +json_credentials='{ "type": "service_account", "project_id": .... }' +``` ### HBase Specify the comma-separated list of HBase Thrift servers for clusters in the format of "(name|host:port)": - [hbase] - hbase_clusters=(Cluster|localhost:9090) +```ini +[hbase] +hbase_clusters=(Cluster|localhost:9090) +``` #### Impersonation doAs Impersonation provides a flexible way to use the same client to impersonate multiple principals. doAs is supported only in Thrift 1. Enable doAs support by adding the following properties to hbase-site.xml on each Thrift gateway: - - hbase.regionserver.thrift.http - true - - - hbase.thrift.support.proxyuser - true - +```xml + + hbase.regionserver.thrift.http + true + + + hbase.thrift.support.proxyuser + true + +``` And the Hue hosts, or * to authorize from any host: - - hadoop.proxyuser.hbase.hosts - * - - - hadoop.proxyuser.hbase.groups - * - +```xml + + hadoop.proxyuser.hbase.hosts + * + + + hadoop.proxyuser.hbase.groups + * + +``` -Note: If you use framed transport, you cannot use doAs impersonation, because SASL does not work with Thrift framed transport. +> **Note:** If you use framed transport, you can't use doAs impersonation, because SASL does not work with Thrift framed transport. #### Kerberos cluster -In a secure cluster its also needs these properties: +In a secure cluster, you also need these properties: - - hbase.thrift.spnego.principal - HTTP/@REALM - Note, _HOST does not work. ## Can be found from CM-Administration->Security->Kerberos Credentials (HTTP/....) - - - hbase.thrift.spnego.keytab.file - hbase.keytab/value> - +```xml + + hbase.thrift.spnego.principal + HTTP/@REALM + + + hbase.thrift.spnego.keytab.file + hbase.keytab + +``` And from the HBase shell, authorize some ends users, e.g. to give full access to `admin`: - hbase shell> grant 'admin', 'RWXCA' +```bash +hbase shell> grant 'admin', 'RWXCA' +``` ## Metadata ### Apache Atlas -In the `[metadata]` section, Hue is supporting Cloudera Navigator and Apache Atlas in order to enrich the [data catalog](/user/browsing/#data-catalog). +In the `[metadata]` section, Hue supports Apache Atlas and Cloudera Navigator to enrich the [data catalog](/user/browsing/#data-catalogs). - [metadata] - [[catalog]] - # The type of Catalog: Apache Atlas, Cloudera Navigator... - interface=atlas - # Catalog API URL (without version suffix). - api_url=http://localhost:21000/atlas/v2 +```ini +[metadata] +[[catalog]] +# The type of Catalog: Apache Atlas, Cloudera Navigator... +interface=atlas +# Catalog API URL (without version suffix). +api_url=http://localhost:21000/atlas/v2 - # Username of the CM user used for authentication. - ## server_user=hue - # Password of the user used for authentication. - server_password= +# Username used for authentication. +## server_user=hue +# Password of the user used for authentication. +server_password= - # Limits found entities to a specific cluster. When empty the entities from all clusters will be included in the search results. - ## search_cluster= +# Limits found entities to a specific cluster. When empty the entities from all clusters will be included in the search results. +## search_cluster= - # Set to true when authenticating via kerberos instead of username/password - ## kerberos_enabled=false +# Set to true when authenticating via kerberos instead of username/password +## kerberos_enabled=false +``` ![Data Catalog Search](https://cdn.gethue.com/uploads/2019/06/SearchWithType_field_name.png) ### Cloudera Navigator -The integration was replaced with Apache Atlas but can still be used. +The Apache Atlas integration replaces this one, which is deprecated but still available in the `[[navigator]]` section. ### Cloudera Navigator Optimizer -The integration is powering the [Risk Alerts and Popular Values](/user/querying/#query-troubleshooting) in the SQL Autocomplete. +The integration powers the [Risk Alerts and Popular Values](/user/querying/#query-troubleshooting) in the SQL Autocomplete. ![Popular joins suggestion](https://cdn.gethue.com/uploads/2017/07/hue_4_query_joins.png) @@ -1160,233 +1394,269 @@ The integration is powering the [Risk Alerts and Popular Values](/user/querying/ ### Apache Spark -This connector leverage the [Apache Livy REST Api](https://livy.incubator.apache.org/). +This connector uses the [Apache Livy REST Api](https://livy.incubator.apache.org/). In the `[[interpreters]]` section: - [[[pyspark]]] - name=PySpark - interface=livy +```ini +[[[pyspark]]] +name=PySpark +interface=livy - [[[sparksql]]] - name=SparkSql - interface=livy +[[[sparksql]]] +name=SparkSql +interface=livy - [[[spark]]] - name=Scala - interface=livy +[[[spark]]] +name=Scala +interface=livy - [[[r]]] - name=R - interface=livy +[[[r]]] +name=R +interface=livy +``` In the `[spark]` section: - [spark] - # The Livy Server URL. - livy_server_url=http://localhost:8998 +```ini +[spark] +# The Livy Server URL. +livy_server_url=http://localhost:8998 +``` -And if using Cloudera distribution, make sure you have notebooks enabled: +Make sure the Notebook app is not blocked and is shown in the menu: - [desktop] - app_blacklist= +```ini +[desktop] +app_blacklist= - [notebook] - show_notebooks=true +[notebook] +show_notebooks=true +``` **YARN: Spark session could not be created** -If seeing an error similar to this with `primitiveMkdir`: +If you see an error similar to this one with `primitiveMkdir`: - The Spark session could not be created in the cluster: at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) - at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) - at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.mkdirs(Unknown Source) - at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2333) ... 20 more - 19/05/13 12:27:07 INFO util.ShutdownHookManager: Shutdown hook called 19/05/13 12:27:07 INFO util.ShutdownHookManager: - Deleting directory /tmp/spark-0d045154-77a0-4e12-94b2-2df18725a4ae YARN Diagnostics: +```text +The Spark session could not be created in the cluster: at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) +at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) +at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) +at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.mkdirs(Unknown Source) +at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2333) ... 20 more +19/05/13 12:27:07 INFO util.ShutdownHookManager: Shutdown hook called 19/05/13 12:27:07 INFO util.ShutdownHookManager: +Deleting directory /tmp/spark-0d045154-77a0-4e12-94b2-2df18725a4ae YARN Diagnostics: +``` -Does your logged-in user have a home dir on HDFS (i.e. `/user/bob`)? (you should see the full error in the Livy or YARN logs). +Check that your logged-in user has a home directory on HDFS, for example `/user/bob`. The Livy or YARN logs show the full error. -In Hue admin for you user, you can click the 'Create home' checkbox and save. +In the Hue user admin, edit the user, select **Create home directory** and save. **CSRF** -Livy supports a configuration parameter in the Livy conf: +Livy has a CSRF protection setting in its configuration: - livy.server.csrf-protection.enabled +```text +livy.server.csrf-protection.enabled +``` -...which is false by default. Upon trying to launch a Livy session from the notebook, Hue will pass along the connection error from Livy as a 400 response that the "Missing Required Header for CSRF protection". To enable it, add to the Hue config: +It is `false` by default. If you enable it in Livy, launching a Livy session from the Editor fails with a 400 response: "Missing Required Header for CSRF protection". To fix it, add this to the Hue configuration: - [spark] - # Whether Livy requires client to use csrf protection. - ## csrf_enabled=false +```ini +[spark] +# Whether Livy requires client to use csrf protection. +csrf_enabled=true +``` **Impersonation** -Let’s say we want to create a shell running as the user bob, this is particularly useful when multi users are sharing a Notebook server +To create a shell that runs as the user `bob`, send this request to Livy. This is useful when multiple users share a Notebook server: - curl -X POST --data '{"kind": "pyspark", "proxyUser": "bob"}' -H "Content-Type: application/json" localhost:8998/sessions +```bash +curl -X POST --data '{"kind": "pyspark", "proxyUser": "bob"}' -H "Content-Type: application/json" localhost:8998/sessions +``` - {"id":0,"state":"starting","kind":"pyspark","proxyUser":"bob","log":[]} +Livy returns: -Do not forget to add the user running Hue (your current login in dev or hue in production) in the Hadoop proxy user list (/etc/hadoop/conf/core-site.xml): +```json +{"id":0,"state":"starting","kind":"pyspark","proxyUser":"bob","log":[]} +``` - - hadoop.proxyuser.hue.hosts - * - - - hadoop.proxyuser.hue.groups - * - +Add the user running Hue (your login in development, `hue` in production) to the Hadoop proxy user list in `/etc/hadoop/conf/core-site.xml`: + +```xml + + hadoop.proxyuser.hue.hosts + * + + + hadoop.proxyuser.hue.groups + * + +``` ### Apache Pig -Pig is native to Hue and depends on the [Oozie service](/administrator/configuration/connectors/#apache-oozie) to be configured: +Pig is native to Hue and depends on the [Oozie service](#apache-oozie) being configured: - [[[pig]]] - name=Pig - interface=oozie +```ini +[[[pig]]] +name=Pig +interface=oozie +``` ### Apache Oozie -In oder to schedule workflows, the `[liboozie]` section of the configuration file: +To schedule workflows, set the Oozie URL in the `[liboozie]` section: - [liboozie] - oozie_url=http://oozie-server.com:11000/oozie +```ini +[liboozie] +oozie_url=http://oozie-server.com:11000/oozie +``` Make sure that the [Share Lib](https://oozie.apache.org/docs/5.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation) is installed. -To configure Hue as a default proxy user, add the following properties to /etc/oozie/conf/oozie-site.xml: +To configure Hue as a default proxy user, add the following properties to `/etc/oozie/conf/oozie-site.xml`: - - - oozie.service.ProxyUserService.proxyuser.hue.hosts - * - - - oozie.service.ProxyUserService.proxyuser.hue.groups - * - +```xml + + + oozie.service.ProxyUserService.proxyuser.hue.hosts + * + + + oozie.service.ProxyUserService.proxyuser.hue.groups + * + +``` ### Apache YARN Hue supports one or two Yarn clusters (two for HA). These clusters should be defined under the `[[[default]]]` and `[[[ha]]]` sub-sections. - # Configuration for YARN (MR2) - # ------------------------------------------------------------------------ - [[yarn_clusters]] +```ini +# Configuration for YARN (MR2) +# ------------------------------------------------------------------------ +[[yarn_clusters]] - [[[default]]] +[[[default]]] - resourcemanager_host=yarn-rm.com - resourcemanager_api_url=http://yarn-rm.com:8088/ - proxy_api_url=http://yarn-proxy.com:8088/ - resourcemanager_port=8032 - history_server_api_url=http://yarn-rhs-com:19888/ +resourcemanager_host=yarn-rm.com +resourcemanager_api_url=http://yarn-rm.com:8088/ +proxy_api_url=http://yarn-proxy.com:8088/ +resourcemanager_port=8032 +history_server_api_url=http://yarn-rhs-com:19888/ +``` ### Apache Sentry -To have Hue point to a Sentry service and another host, modify these hue.ini properties: - - [libsentry] - # Hostname or IP of server. - hostname=localhost +To point Hue to a Sentry service, set these hue.ini properties: - # Port the sentry service is running on. - port=8038 +```ini +[libsentry] +# Hostname or IP of server. +hostname=localhost - # Sentry configuration directory, where sentry-site.xml is located. - sentry_conf_dir=/etc/sentry/conf +# Port the sentry service is running on. +port=8038 -Hue will also automatically pick up the server name of HiveServer2 from the sentry-site.xml file of /etc/hive/conf. +# Sentry configuration directory, where sentry-site.xml is located. +sentry_conf_dir=/etc/sentry/conf +``` -![Listing of Sentry Tables privileges](https://cdn.gethue.com/uploads/2019/04/HueSecurityRoles.png +Hue also picks up the HiveServer2 server name from the `sentry-site.xml` file in `/etc/hive/conf`. -And that’s it, you can know specify who can see/do what directly in a Web UI! The app sits on top of the standard Sentry API and so it fully compatible with Sentry. Next planned features will bring Solr Collections, HBase privilege management as well as more bulk operations and a tighter integration with HDFS. +![Listing of Sentry Tables privileges](https://cdn.gethue.com/uploads/2019/04/HueSecurityRoles.png) +Users can then manage privileges in the Hue web UI. The app uses the standard Sentry API. -To be able to edit roles and privileges in Hue, the logged-in Hue user needs to belong to a group in Hue that is also an admin group in Sentry (whatever UserGroupMapping Sentry is using, the corresponding groups must exist in Hue or need to be entered manually). For example, our ‘hive’ user belongs to a ‘hive’ group in Hue and also to a ‘hive’ group in Sentry: +To edit roles and privileges in Hue, the logged-in Hue user must belong to a group in Hue that is also an admin group in Sentry. Whatever group mapping Sentry uses, the corresponding groups must exist in Hue or be entered manually. For example, the `hive` user belongs to a `hive` group in Hue and to a `hive` group in Sentry: - - sentry.service.admin.group - hive,impala,hue - +```xml + + sentry.service.admin.group + hive,impala,hue + +``` -**Notes** +Follow these steps: * Create a role in the Sentry app through Hue * Grant privileges to that role such that the role can see the database in the Sentry app * Create a group in Hue with the same name as the role in Sentry * Grant that role to a user in Hue -* Ensure that the user in Hue has an equivalent O/S level -* Ensure a user has an O/S level account on all hosts and that user is part of a group with the same name as the group in Hue (this assumes that the default ShellBasedUnixGroupsMapping is set for HDFS in CM) - -Our users are: - -* hive (admin) belongs to the hive group -* user1_1 belongs to the user_group1 group -* user2_1 belongs to the user_group2 group - -We synced the Unix users/groups into Hue with these commands: - - export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk '{print $9}'`" - - build/env/bin/hue useradmin_sync_with_unix --min-uid=1000 - -If using the package version and has the CDH repository register, install sentry with: - - sudo apt-get install sentry - -If using Kerberos, make sure ‘hue’ is allowed to connect to Sentry in /etc/sentry/conf/sentry-site.xml: - - - sentry.service.allow.connect - impala,hive,solr,hue - - -Here is an example of sentry-site.xml - - - - - sentry.service.security.mode - none - - - sentry.service.admin.group - hive,romain - - - sentry.service.allow.connect - impala,hive,solr - - - sentry.store.jdbc.url - jdbc:derby:;databaseName=sentry_store_db;create=true - - - sentry.store.jdbc.driver - org.apache.derby.jdbc.EmbeddedDriver - - - sentry.store.jdbc.password - aaa - - +* Ensure that the user has an operating system account on all hosts, and that the account belongs to a group with the same name as the group in Hue. This assumes that HDFS uses the default `ShellBasedUnixGroupsMapping`. + +In this example, the users are: + +* `hive` (admin) belongs to the `hive` group +* `user1_1` belongs to the `user_group1` group +* `user2_1` belongs to the `user_group2` group + +Sync the Unix users and groups into Hue with this command: + +```bash +./build/env/bin/hue useradmin_sync_with_unix --min-uid=1000 +``` + +If you use Kerberos, make sure the `hue` user can connect to Sentry in `/etc/sentry/conf/sentry-site.xml`: + +```xml + + sentry.service.allow.connect + impala,hive,solr,hue + +``` + +Here is an example `sentry-site.xml`: + +```xml + + + + sentry.service.security.mode + none + + + sentry.service.admin.group + hive,romain + + + sentry.service.allow.connect + impala,hive,solr + + + sentry.store.jdbc.url + jdbc:derby:;databaseName=sentry_store_db;create=true + + + sentry.store.jdbc.driver + org.apache.derby.jdbc.EmbeddedDriver + + + sentry.store.jdbc.password + aaa + + +``` ### Apache Knox - [[knox]] +To let Hue trust requests that come through Apache Knox, set these properties: + +```ini +[desktop] +[[knox]] + +# This is a list of hosts that knox proxy requests can come from +## knox_proxyhosts=server1.domain.com,server2.domain.com - # This is a list of hosts that knox proxy requests can come from - ## knox_proxyhosts=server1.domain.com,server2.domain.com +# List of Kerberos principal name which is allowed to impersonate others +## knox_principal=knox1,knox2 - # List of Kerberos principal name which is allowed to impersonate others - ## knox_principal=knox1,knox2 +# Comma separated list of strings representing the ports that the Hue server can trust as knox port. +## knox_ports=80,8443 +``` - # Comma separated list of strings representing the ports that the Hue server can trust as knox port. - ## knox_ports=80,8443 diff --git a/docs/docs-site/content/administrator/configuration/server/_index.md b/docs/docs-site/content/administrator/configuration/server/_index.md index b6bed675e0b..a0d5d0bea1e 100644 --- a/docs/docs-site/content/administrator/configuration/server/_index.md +++ b/docs/docs-site/content/administrator/configuration/server/_index.md @@ -5,821 +5,807 @@ draft: false weight: 1 --- -This section is about configuring the Hue server itself. -These configuration variables are under the `[desktop]` section in -the `conf/hue.ini` configuration file. +This page explains how to configure the Hue server itself. +Unless stated otherwise, the properties below go under the `[desktop]` section of the `hue.ini` file. ## Basics -### Point to MySQL or Postgres +### Point to MySQL or PostgreSQL -Directly below the `[[database]]` line, add the following options (and modify accordingly for -your MySQL setup): +Hue stores users and saved queries in a database. To use MySQL or PostgreSQL instead of the default SQLite, add the following options directly below the `[[database]]` line (under `[desktop]`). Change them to match your setup: +```ini +[desktop] + [[database]] host=localhost port=3306 engine=mysql user=hue password=secretpassword name=hue +``` -And run the table creation one time: +For PostgreSQL, set `engine=postgresql_psycopg2` and use your PostgreSQL port. +For MariaDB, use the `mysql` engine. - ./build/env/bin/hue migrate +Then create the tables. You only need to do this once: -### Specifying the HTTP port +```bash +./build/env/bin/hue migrate +``` -Hue uses CherryPy web server. You can use the following options to -change the IP address and port that the web server listens on. -The default setting is port 8888 on all configured IP addresses. +See [Configuring with another database](/administrator/administration/database/#configure-another-database) for the full steps, including how to migrate existing data. - # Webserver listens on this address and port - http_host=0.0.0.0 - http_port=8888 +### Specify the HTTP port + +Use the following options to change the IP address and port that the web server listens on. +The default is port 8888 on all configured IP addresses. + +```ini +[desktop] + # Webserver listens on this address and port + http_host=0.0.0.0 + http_port=8888 +``` ### Gunicorn server -Instead of running CherryPy, [Gunicorn](https://gunicorn.org/) will be launched: +Hue serves requests with [Gunicorn](https://gunicorn.org/) by default. You can tune it with these options: + +```ini +[desktop] + # Gunicorn worker class: gevent, eventlet, gthread or sync. + gunicorn_work_class=gthread + + # Number of Gunicorn worker processes. + gunicorn_number_of_workers=1 + + # Workers silent for more than this many seconds are killed and restarted. + gunicorn_worker_timeout=900 - # Set to true to use CherryPy as the webserver, set to false - # to use Gunicorn as the webserver. Defaults to CherryPy if - # key is not specified. - use_cherrypy_server=false + # After a restart signal, workers have this many seconds to finish their requests. + gunicorn_worker_graceful_timeout=900 +``` - # Gunicorn work class: gevent or evenlet, gthread or sync. - gunicorn_work_class=sync +> **Note**: `use_cherrypy_server` is still a valid option, but CherryPy is no longer part of the Hue Python requirements. On Python 3 the default is `false`: keep it and use Gunicorn. -### Specifying the Secret Key +### Specify the secret key -For security, you should also specify the secret key that is used for secure -hashing in the session store. Enter a long series of random characters -(30 to 60 characters is recommended). +Set a secret key to protect the hashing of the session store. Use a long series of random characters (30 to 60 characters is recommended). - secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn **Note**: If you don't set a secret key, your session cookies are not secure. Hue runs, but it displays error messages telling you to set the key. -In the Hue ini configuration file, in the [desktop] section, you can enter the names of the app to hide: +### Disable some apps - [desktop] - # Comma separated list of apps to not load at server startup. - app_blacklist=beeswax,impala,security,filebrowser,jobbrowser,rdbms,jobsub,pig,hbase,sqoop,zookeeper,metastore,spark,oozie,indexer +To hide apps, list their names in `app_blacklist`: + +```ini +[desktop] + # Comma separated list of apps to not load at server startup. + app_blacklist=pig,hbase,sqoop,zookeeper,oozie +``` [Read more about it here](http://gethue.com/mini-how-to-disabling-some-apps-from-showing-up/). ## Authentication -By default (`AllowFirstUserDjangoBackend`), the first user who logs in to Hue can choose any -username and password and becomes an administrator automatically. This -user can create other user and administrator accounts. User information is -stored in the Django database in the Django backend. +By default, Hue uses the `AllowFirstUserDjangoBackend` backend. The first user who logs in can choose any username and password and becomes an administrator automatically. This user can create other users and administrators. Hue stores user information in its database. + +The authentication system is pluggable. You select one or more backends with the `backend` property in the `[[auth]]` section. These are the main options. -The authentication system is pluggable. Here is a list of some of the possible authentications: +### Username and password -### Username / Password +This is the default Hue backend. It makes the first user who logs in the superuser. After that, it relies on Django and the user manager to authenticate users. -This is the default Hue backend. It creates the first user that logs in as the super user. After this, it relies on Django and the user manager to authenticate users. +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.AllowFirstUserDjangoBackend +``` - desktop.auth.backend.AllowFirstUserDjangoBackend +### Allow all -### Allow All +This backend does not require a password. Hue authenticates every user automatically and uses the username that is provided. -This backend does not require a password for users to log in. All users are automatically authenticated and the username is set to what is provided. +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.AllowAllBackend +``` - desktop.auth.backend.AllowAllBackend +> **Warning**: Do not use this backend on a server that others can reach. ### LDAP -Authenticates users against an LDAP service. +This backend authenticates users against an LDAP service. + +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.LdapBackend +``` + +There are two ways to bind Hue to an LDAP directory service: - desktop.auth.backend.LdapBackend +* Search bind: Hue searches for the user with a search base (and optionally an attribute and a filter), then binds with the DN it found. +* Direct bind: Hue authenticates without searching, in one of two ways: + * NT domain: bind to Microsoft Active Directory with `username@domain` (the UPN). + * Username pattern: bind to an open standard LDAP server with the full path in the directory information tree (DIT). -There are two ways to bind Hue with an LDAP directory service: +> **Note**: The username pattern does not work with Active Directory (AD), because AD inserts spaces into the UID, which Hue cannot process. -* Search Bind: Hue searches for user credentials with search base (and attribute and filter). -* Direct Bind: Hue authenticates (without searching) in one of two ways: - * NT Domain: Bind to Microsoft Active Directory with username@domain (the UPN)or - * Username Pattern: Bind to open standard LDAP with full path of directory information tree (DIT). +To avoid sending credentials in the clear, encrypt the connection. Use LDAP over SSL (LDAPS) on port 636 by default, or the StartTLS extension on the standard LDAP port (389 by default). LDAPS is recommended. In both cases you need a CA certificate. -Note: Username pattern does not work with AD because AD inserts spaces into the UID which Hue cannot process. +The following table lists the supported combinations of authentication method and encryption. -Encryption: To prevent credentials from transmitting in the clear, encrypt with LDAP over SSL, using the LDAPS protocol on the LDAPS port (636 by default); or encrypt with the StartTLS extension using the standard LDAP protocol and port (389 by default). Cloudera recommends LDAPS. You must have a CA Certificate in either case. +| LDAP auth action | Encrypted (LDAPS) | Encrypted (LDAP+TLS) | Not encrypted (LDAP) | +| -------------------------- | ----------------- | -------------------- | -------------------- | +| Search bind | AD, LDAP | AD, LDAP | AD, LDAP | +| Direct bind - NT domain | AD | AD | AD | +| Direct bind - user pattern | LDAP | LDAP | LDAP | -Hue Supported LDAP Authentication and Encryption Methods +#### Search bind - LDAP Auth Action Encrypted (LDAPS) Encrypted (LDAP+TLS) Not Encrypted (LDAP) - Search Bind AD, LDAP AD, LDAP AD, LDAP - Direct Bind - NT Domain AD AD AD - Direct Bind - User Pattern LDAP LDAP LDAP +Search bind does an `ldapsearch` against one or more directory services and binds with the DN it finds and the user's password. Hue searches the subtree below the base DN. If `user_name_attr` is set, Hue looks for an entry whose attribute has the same value as the short name entered at login. -Example of a Search Bind configuration encrypted with LDAPS: +> **Note**: Search bind works with all directory service types. It is also the only method that can synchronize groups at login (set `sync_groups_on_login`). - [[custom]] - [[auth]] +Example of a search bind configuration encrypted with LDAPS: + +```ini +[desktop] + [[auth]] backend=desktop.auth.backend.LdapBackend - [[ldap]] - ldap_url=ldaps://w2k8-1.ad.sec.cloudera.com:636 + [[ldap]] + ldap_url=ldaps://ldap.example.com:636 search_bind_authentication=true - ldap_cert=//w2k8-1-root.pem + ldap_cert=//ca-root.pem use_start_tls=false create_users_on_login=true - base_dn="DC=ad,DC=sec,DC=cloudera,DC=com" - bind_dn="@ad.sec.cloudera.com" + base_dn="DC=ad,DC=example,DC=com" + bind_dn="@ad.example.com" bind_password_script=/ test_ldap_user="testuser1" test_ldap_group="testgroup1" [[[users]]] - user_filter="objectclass=user" - user_name_attr="sAMAccountName" + user_filter="objectclass=user" + user_name_attr="sAMAccountName" [[[groups]]] - group_filter="objectclass=group" - group_name_attr="cn" - group_member_attr="member" - -Example of a Direct Bind configuration for Active Directory encrypted with LDAPS: - - [[ldap]] - ldap_url=ldaps://w2k8-1.ad.sec.cloudera.com:636 - search_bind_authentication=false - nt_domain=ad.sec.cloudera.com - ldap_cert=//w2k8-1-root.pem - use_start_tls=false - create_users_on_login=true - base_dn="DC=ad,DC=sec,DC=cloudera,DC=com" - bind_dn="" - bind_password_script=/ - ... - -Example of a Direct Bind configuration for Active Directory encrypted with StartTLS: + group_filter="objectclass=group" + group_name_attr="cn" + group_member_attr="member" +``` - [[ldap]] - ldap_url=ldap://w2k8-1.ad.sec.cloudera.com:389 - search_bind_authentication=false - nt_domain=ad.sec.cloudera.com - ldap_cert=/opt/cloudera/security/cacerts/w2k8-1-root.pem - use_start_tls=true - create_users_on_login=true - base_dn="DC=ad,DC=sec,DC=cloudera,DC=com" - bind_dn="cconner" - bind_password_script=/ - ... - - -#### Search Bind - -Search bind authentication does an ldapsearch against one or more directory services and binds with the found distinguished name (DN) and password. Hue searches the subtree from the base distinguished name. If LDAP Username Attribute is set, Hue looks for an entry whose attribute has the same value as the short name given at login. - -Important: Search binding works with all directory service types. It is also the only method that allows synchronizing groups at login (set with sync_groups_on_login in a safety-valve). - -Set the following required properties: - -Authentication Backend desktop.auth.backend.LdapBackend - - LDAP URL ldaps://:636 (or ldap://:389) - LDAP Server CA Certificate /path_to_certificate/cert.pem - LDAP Search Base DC=mycompany,DC=com - LDAP Bind User Distinguished Name username@domain - LDAP Bind Password bind_user_password - Use Search Bind Authentication TRUE - Enable LDAP TLS FALSE if using LDAPS or not encrypting - Create LDAP users on login TRUE +The main properties are: -Note: To encrypt with TLS, set LDAP URL to ldap://:389 and check Enable LDAP TLS. For a proof of concept without encryption, use ldap://:389, remove the value for LDAP Server CA Certificate, and uncheck Enable LDAP TLS. +| Property | Value | +| ---------------------------- | ---------------------------------------------------------------------- | +| `ldap_url` | `ldaps://:636` (or `ldap://:389`) | +| `ldap_cert` | Path to the CA certificate of the LDAP server, for example `/path_to_certificate/cert.pem` | +| `base_dn` | The search base, for example `DC=mycompany,DC=com` | +| `bind_dn` | The bind user, in the form `username@domain` | +| `bind_password_script` | Script that prints the bind user password (see [Store passwords in a script](#store-passwords-in-a-script)) | +| `search_bind_authentication` | `true` | +| `use_start_tls` | `false` for LDAPS or no encryption, `true` for StartTLS | +| `create_users_on_login` | `true` | -You can optionally improve search performance with attributes and filters. +> **Note**: To encrypt with StartTLS, set `ldap_url` to `ldap://:389` and `use_start_tls=true`. For a proof of concept without encryption, use `ldap://:389`, remove `ldap_cert`, and set `use_start_tls=false`. - LDAP User Filter objectclass=user (default = *) - LDAP Username Attribute sAMAccountName (AD default), uid (LDAP default) - LDAP Group Filter objectclass=group (default = *) - LDAP Group Name Attribute cn (default) - LDAP Group Membership Attribute member (default) +You can optionally improve the search performance with filters and attributes: -Note: With the user settings in the table above, the LDAP search filter has the form: (&(objectClass=user)(sAMAccountName=)). +| Property | Value | +| ------------------- | ---------------------------------------------------------- | +| `user_filter` | For example `objectclass=user` | +| `user_name_attr` | `sAMAccountName` for AD, `uid` for most other LDAP servers | +| `group_filter` | For example `objectclass=group` | +| `group_name_attr` | `cn` | +| `group_member_attr` | `member` | -Add any valid user and/or valid group to quickly test your LDAP configuration. +> **Note**: With these user settings, the LDAP search filter looks like `(&(objectClass=user)(sAMAccountName=))`. - LDAP Username for Test LDAP Configuration Any valid user - LDAP Group Name for Test LDAP Configuration Any valid group +To test your configuration, set `test_ldap_user` and `test_ldap_group` to a valid user and a valid group. -Note: The syntax of Bind Distinguished Name differs per bind method: +> **Note**: The syntax of `bind_dn` depends on the bind method. +> +> * Search bind: `username@domain` +> * Direct bind with NT domain: `username` +> * Direct bind with username pattern: a DN string (the full DIT path) - Search Bind: username@domain - Direct Bind with NT Domain: username - Direct Bind with Username Pattern: DN string (full DIT path) +Skip `bind_dn` and `bind_password_script` if your server supports anonymous binding. -Do not use if anonymous binding is supported. +You can test your settings with `ldapsearch` on the command line: - ## You can test ldapsearch at the command line as follows: - LDAPTLS_CACERT=// ldapsearch -H ldaps://:636 \ - -D "" -w -b "samaccountname=" +```bash +LDAPTLS_CACERT=// ldapsearch -H ldaps://:636 \ + -D "" -w -b "samaccountname=" +``` -Note: To run ldapsearch with a CA certificate, you may need to install ldap_utils on Debian/Ubuntu and openldap-clients on RHEL/CentOS. +> **Note**: To run `ldapsearch` with a CA certificate, you might need to install `ldap-utils` on Debian/Ubuntu or `openldap-clients` on RHEL/CentOS. -#### Direct Bind +#### Direct bind -To authenticate with direct binding, Hue needs either the User Principal Name (UPN) for Active Directory, or the full path to the LDAP user in the Directory Information Tree (DIT) for open standard LDAP. +For direct bind, Hue needs either the user principal name (UPN) for Active Directory, or the full path to the LDAP user in the DIT for open standard LDAP. -Important: Direct binding only works with one domain. For multiple directories, use Search Bind. +> **Note**: Direct bind only works with one domain. For multiple directories, use search bind. -To directly bind to an Active Directory/LDAP server with NT domain: +To bind directly to an Active Directory server with an NT domain, configure the properties like a search bind, with these changes: -Click the Configuration tab and filter by scope=Service-wide and category=Security. +* Set `nt_domain` to your NT domain. +* Set `bind_dn` to the username only (not `username@domain`). +* Set `search_bind_authentication=false`. -Set LDAP properties exactly like Search Bind with these exceptions: - Active Directory Domain - LDAP Bind User Distinguished Name (not username@domain) - Use Search Bind Authentication FALSE +```ini +[desktop] + [[ldap]] + ldap_url=ldaps://ldap.example.com:636 + search_bind_authentication=false + nt_domain=ad.example.com + ldap_cert=//ca-root.pem + use_start_tls=false + create_users_on_login=true + base_dn="DC=ad,DC=example,DC=com" + bind_dn="" + bind_password_script=/ +``` -Test your LDAP configuration, and when successful, Restart Hue. -To directly bind to an open standard LDAP server with a username pattern: +To use StartTLS instead of LDAPS, set `ldap_url=ldap://ldap.example.com:389` and `use_start_tls=true`. -Remove the value for Active Directory Domain. +To bind directly to an open standard LDAP server with a username pattern: -Set both LDAP Username Pattern and LDAP Bind User Distinguished Name to a DN string that represents the full path of the directory information tree, from UID to top level domain. +* Leave `nt_domain` empty. +* Set `ldap_username_pattern` to a DN string that represents the full path in the DIT, from the UID up to the top-level domain. +* Set `bind_dn` to a DN string in the same format. -Note: When using direct bind, set LDAP Search Base, not for authentication (you can log on to Hue without it), but to Synchronize Hue with LDAP Server. +> **Note**: With direct bind, set `base_dn` anyway. You can log in to Hue without it, but you need it to synchronize Hue with the LDAP server. -#### Troubleshooting +Test your LDAP configuration, then restart Hue. -##### Issue: -Hue is not able to sync the LDAP group in Hue and it reports the following error : Could not get LDAP details for groups in pattern +#### Troubleshooting LDAP -##### Solution: -check your configuration if 'subgroups' is set under [[ldap]] and remove it. +**Issue**: Hue cannot synchronize an LDAP group and reports `Could not get LDAP details for groups in pattern `. +**Solution**: Check your configuration. If `subgroups` is set under `[[ldap]]`, remove it. ### SAML -Secure Assertion Markup Language (SAML) single sign-on (SSO) backend. Delegates authentication to the configured Identity Provider. See Configuring Hue for SAML for more details. +The Security Assertion Markup Language (SAML) single sign-on (SSO) backend delegates authentication to the Identity Provider you configure. - libsaml.backend.SAML2Backend - - -Authenticate Hue Users with SAML - -Hue supports SAML (Security Assertion Markup Language) for Single Sign-on (SSO) authentication. +```ini +[desktop] + [[auth]] + backend=libsaml.backend.SAML2Backend +``` The SAML 2.0 Web Browser SSO profile has three components: -* User Agent - Browser that represents you, the user, seeking resources. -* Service Provider (SP) - Service (Hue) that sends authentication requests to SAML. -* Identity Provider (IdP) - SAML service that authenticates users. +* User Agent: the browser that represents you, the user, requesting resources. +* Service Provider (SP): the service (Hue) that sends authentication requests to SAML. +* Identity Provider (IdP): the SAML service that authenticates users. When a user requests access to an application, the Service Provider (Hue) sends an authentication request from the User Agent (browser) to the Identity Provider. The Identity Provider authenticates the user, sends a response, and redirects the browser back to Hue. -This page explains how to configure Hue, the Service Provider, and gives guidance on how to configure the Identity Provider, which differs per product. -Configure Hue for SAML Authentication +This section explains how to configure Hue as the Service Provider. How you configure the Identity Provider differs per product. -The Service Provider (Hue) and the Identity Provider use a metadata file to confirm each other's identity. Hue stores metadata from the SAML server, and the IdP stores metadata from Hue server. +#### Configure Hue for SAML -In Configure Hue at the Command Line, you must copy the metadata from your IdP's SAML server and store it in an XML file on every host with a Hue server. -Important: Read the documentation of your Identity Provider for details on how to procure the XML of the SAML server metadata. +The Service Provider (Hue) and the Identity Provider use a metadata file to confirm each other's identity. Hue stores the metadata from the SAML server, and the IdP stores the metadata from the Hue server. -Configure Hue at the Command Line -Important: You may need to disable cipher algorithms. See SAML SSL Error in Troubleshooting below. +1. Install the build libraries on every host that runs a Hue server: -Install the following libraries on all hosts in your cluster: + ```bash + # RHEL/CentOS + yum install git gcc python3-devel swig openssl + # Ubuntu/Debian + apt-get install git gcc python3-dev swig openssl + # SLES + zypper install git gcc python3-devel swig openssl make libxslt-devel libltdl-devel + ``` - ## RHEL/CentOS - yum install git gcc python-devel swig openssl - ## Ubuntu/Debian - apt-get install git gcc python-dev swig openssl - ## SLES - zypper install git gcc python-devel swig openssl make libxslt-devel libltdl-devel +2. Install `xmlsec1` on every host that runs a Hue server. Make sure the user that runs Hue can execute it. -Install xmlsec1 and xmlsec1-openssl on all hosts in the cluster: + ```bash + # RHEL/CentOS + yum install xmlsec1 xmlsec1-openssl + # Ubuntu/Debian + apt-get install xmlsec1 libxmlsec1-openssl + ``` -Important: Ensure that the xmlsec1 package is executable by the user, hue. +3. Copy the metadata from your IdP's SAML server and save it as an XML file on every host with a Hue server. Read the documentation of your Identity Provider to find how to get this metadata. For example, with Shibboleth, open `https://:8443/idp/shibboleth`, copy the metadata content, and paste it into an `.xml` file. - ## RHEL/CentOS - yum install xmlsec1 xmlsec1-openssl - Note: If xmlsec libraries are not available, use the appropriate epel repository: - ## For RHEL/CentOS 7 - wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-6.noarch.rpm - rpm -ivh epel-release-7-6.noarch.rpm - ## Ubuntu/Debian - apt-get install xmlsec1 libxmlsec1-openssl - ## SLES (get latest version) - wget http://www.aleksey.com/xmlsec/download/xmlsec1-1.2.24.tar.gz - tar -xvzf xmlsec1-1.2.24.tar.gz - cd xmlsec1-1.2.24 - ./configure && make - make install + > **Note**: You might need to edit the copied metadata. For example, the IdP port number (8443) can be missing from its URL. -Copy metadata from your IdP's SAML server and save it as an XML file on every host with a Hue server. -For example, if your Identity Provider is Shibboleth, visit https://:8443/idp/shibboleth, copy the metadata content, and paste it into an .xml file. + ```bash + mkdir -pm 755 /path/to/saml/ + cd /path/to/saml/ + vim idp--metadata.xml + ``` -Note: You may have to edit the copied metadata; for example, the IdP's port number (8443) may be missing from its URL. +4. Set `key_file` and `cert_file` (see the [SAML properties](#saml-properties)). - mkdir -pm 755 /opt/cloudera/security/saml/ - cd /opt/cloudera/security/saml/ - vim idp--metadata.xml + > **Warning**: Set the key and certificate files even if you do not encrypt assertions. Hue checks that these files exist and are valid, even when they are not needed. They cannot be empty files. This is a known issue. -Add key_file and cert_file for encrypted assertions–see Table of SAML Parameters. + If necessary, create dummy files: -Warning: Add key and cert files even if not encrypting assertions. Hue checks for the existence and validity of these files even if they are not needed! They cannot be empty files. This is a known issue. + ```bash + openssl genrsa -des3 -out dummy.key 2048 + openssl rsa -inform PEM -outform PEM -in dummy.key -pubout -out dummy-nopass.pem + ``` -If necessary, create "valid" dummy files: +5. Add the SAML settings to `hue.ini`. Example with OpenAM: - openssl genrsa -des3 -out dummy.key 2048 - openssl rsa -inform PEM -outform PEM -in dummy.key -pubout -out dummy-nopass.pem + ```ini + [desktop] + redirect_whitelist="^\/.*$,^http:\/\/:8080\/.*$" + [[auth]] + backend=libsaml.backend.SAML2Backend -Configure Hue + [libsaml] + xmlsec_binary=/usr/bin/xmlsec1 + metadata_file=/path/to/saml/idp-openam-metadata.xml + key_file=/path/to/saml/host.key + cert_file=/path/to/saml/host.pem + username_source=nameid + name_id_format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient" + entity_id= + logout_enabled=false + ``` -Currently, all hue.ini properties for SAML must be added to Hue Service safety-valve in Cloudera Manager. -Log on to Cloudera Manager and go to Hue > Configuration. + > **Note**: On SLES, the `xmlsec1` binary can be in `/usr/local/bin/`. If so, set `xmlsec_binary=/usr/local/bin/xmlsec1` and add `/usr/local/lib/` to `LD_LIBRARY_PATH` for the Hue process. - ## Example Settings using Open AM: - [desktop] - redirect_whitelist="^\/.*$,^http:\/\/clr.sec.cloudera.com:8080\/.*$" - [[auth]] - backend=libsaml.backend.SAML2Backend - [libsaml] - xmlsec_binary=/usr/bin/xmlsec1 - metadata_file=/opt/cloudera/security/saml/idp-openam-metadata.xml - key_file=/opt/cloudera/security/saml/host.key - cert_file=/opt/cloudera/security/saml/host.pem - username_source=nameid - name_id_format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient" - entity_id= - logout_enabled=false - -Note: For SLES distributions, the xmlsec binary may be in /usr/local/bin/. If so: - -* Set Hue Service Advanced Configuration Snippet: xmlsec_binary=/usr/local/bin/xmlsec1 -* Set Hue Service Environment Advanced Configuration Snippet: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/ - -Integrate IdP SAML Server with Hue - -After Hue is configured and restarted, copy the metadata generated by Hue server and send it to your Identity Provider so they can configure the SAML server. - -Ensure Hue is configured, restarted, and running. -Go to http://:8889/saml2/metadata. - -Copy the metadata and send it to your Identity Provider. - -Ensure that your Identity Provider configures the SAML server with the Hue metadata (just as you configured the Hue server with SAML metadata). - -SAML Properties in hue.ini - -Table of SAML Parameters - -SAML Parameter Description - authn_requests_signed Boolean, that when True, signs Hue-initiated authentication requests with X.509 certificate. - backend Hard-coded value set to SAML backend library packaged with Hue (libsaml.backend.SAML2Backend). - base_url URL that SAML Identity Provider uses for responses. Typically used in Load balanced Hue environments. - cert_file Path to X.509 certificate sent with encrypted metadata. File format must be .PEM. - create_users_on_login Boolean, that when True, creates users from OpenId, upon successful login. - entity_id Service provider ID. Can also accept pattern where '' is replaced with server URL base. - key_file Path to private key used to encrypt metadata. File format must be .PEM. - key_file_password Password used to decrypt the X.509 certificate in memory. - logout_enabled Boolean, that when True, enables single logout. - logout_requests_signed Boolean, that when True, signs Hue-initiated logout requests with an X.509 certificate. - metadata_file Path to readable metadata XML file copied from Identity Provider. - name_id_format Format of NameID that Hue requests from SAML server. - optional_attributes Comma-separated list of optional attributes that Hue requests from Identity Provider. - required_attributes Comma-separated list of required attributes that Hue requests from Identity Provider. For example, uid and email. - redirect_whitelist Fully qualified domain name of SAML server: "^\/.*$,^https:\/\/\/.*$". - user_attribute_mapping Map of Identity Provider attributes to Hue django user attributes. For example, {'uid':'username', 'email':'email'}. - username_source Declares source of username as nameid or attributes. - xmlsec_binary Path to xmlsec_binary that signs, verifies, encrypts/decrypts SAML requests and assertions. Must be executable by user, hue. - Description of some properties to be set in hue.ini (via Cloudera Manager): - redirect_whitelist [desktop] - -Set to the fully qualified domain name of the SAML server so that Hue can redirect to the SAML server for authentication. - - [desktop] - redirect_whitelist=^\/.$,^https:\/\/\/.$ - Note: Hue uses redirect_whitelist to protect itself from redirecting to unapproved URLs. - backend [desktop]>[[auth]] - Point to the SAML backend (packaged with Hue): - backend=libsaml.backend.SAML2Backend - xmlsec_binary [libsaml] - Point to the xmlsec1 library path: - xmlsec_binary=/usr/bin/xmlsec1 - Note: To find the path, run: which xmlsec1 - metadata_file [libsaml] - Point to the path of the XML file you created from the IdP's metadata: - metadata_file=/path/to/.xml - key_file and cert_file [libsaml] - -To encrypt communication between Hue and the Identity Provider, you need a private key and certificate. The private key signs requests sent to the Identity Provider and the certificate file encrypts and decrypts messages from the Identity Provider. +6. Restart Hue. -Copy these files from the Identity Provider and set key_file and cert_file to their respective paths. Both files are in PEM format and must be named with the .PEM extension. +#### Integrate the IdP with Hue -Note: The key and certificate files specified by the key_file and cert_file parameters in hue.ini must be .PEM files. -Users with password-protected certificates can set the property, key_file_password in hue.ini. Hue uses the password to decrypt the SAML certificate in memory and passes it to xmlsec1 through a named pipe. The decrypted certificate never touches the disk. This only works for POSIX-compatible platforms. +After you configure and restart Hue, send the metadata that Hue generates to your Identity Provider so that they can configure the SAML server. -Troubleshooting +1. Make sure Hue is configured, restarted, and running. +2. Open `http://:/saml2/metadata/` (port 8888 by default). +3. Copy the metadata and send it to your Identity Provider. +4. Make sure your Identity Provider configures the SAML server with the Hue metadata, just as you configured Hue with the SAML server metadata. -Remember to Enable DEBUG for logging. +#### SAML properties -SAML SSL Error +The following properties go in the `[libsaml]` section unless stated otherwise. -OpenSSL might fail in CDH 5.5.x and higher with this message: +| Property | Description | +| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `authn_requests_signed` | Boolean. When true, signs Hue-initiated authentication requests with an X.509 certificate. | +| `backend` | In `[desktop]` > `[[auth]]`. Set to `libsaml.backend.SAML2Backend`. | +| `base_url` | URL that the SAML Identity Provider uses for responses. Typically used in load-balanced Hue environments. | +| `cert_file` | Path to the X.509 certificate sent with the encrypted metadata. The file must be in PEM format. | +| `create_users_on_login` | Boolean. When true, creates users on successful login. | +| `entity_id` | Service provider ID. Accepts a pattern where `` is replaced with the server URL base. | +| `key_file` | Path to the private key used to encrypt the metadata. The file must be in PEM format. | +| `key_file_password` | Password used to decrypt the X.509 certificate in memory. You can use `key_file_password_script` instead. | +| `logout_enabled` | Boolean. When true, enables single logout. | +| `logout_requests_signed` | Boolean. When true, signs Hue-initiated logout requests with an X.509 certificate. | +| `metadata_file` | Path to the readable metadata XML file copied from the Identity Provider. | +| `name_id_format` | Format of the NameID that Hue requests from the SAML server. | +| `optional_attributes` | Comma-separated list of optional attributes that Hue requests from the Identity Provider. | +| `required_attributes` | Comma-separated list of required attributes that Hue requests from the Identity Provider. For example, `uid` and `email`. | +| `redirect_whitelist` | In `[desktop]`. Comma-separated regular expressions of allowed redirect URLs, such as `^\/.*$,^https:\/\/\/.*$`. | +| `user_attribute_mapping` | Map of Identity Provider attributes to Hue Django user attributes. For example, `{'uid':'username', 'email':'email'}`. | +| `username_source` | Source of the username: `nameid` or `attributes`. | +| `xmlsec_binary` | Path to the `xmlsec1` binary that signs, verifies, encrypts, and decrypts SAML requests and assertions. The Hue user must be able to execute it. To find the path, run `which xmlsec1`. | - SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_CHECK_CERT_AND_ALGORITHM', 'dh key too small')] +Notes: -To resolve, append the following code to the file, /usr/java/-cloudera/jre/lib/security/java.security: +* `redirect_whitelist` protects Hue from redirecting to unapproved URLs. Set it so that Hue can redirect to your SAML server for authentication. +* To encrypt communication between Hue and the Identity Provider, you need a private key and a certificate. Both files must be in PEM format. +* If your certificate is password protected, set `key_file_password`. Hue uses it to decrypt the certificate in memory and passes it to `xmlsec1` through a named pipe. The decrypted certificate never touches the disk. This only works on POSIX-compatible platforms. - jdk.tls.disabledAlgorithms=MD5, RC4, DH +#### Troubleshooting SAML -SAML Decrypt Error +Enable `DEBUG` logging first. -The following error is an indication that you are using a slightly different SAML protocol from what Hue expects: +**SAML decrypt error** - Error: ('failed to decrypt', -1) +This error means that your Identity Provider uses a slightly different SAML protocol than Hue expects: -To resolve: +```text +Error: ('failed to decrypt', -1) +``` -Download and rename Python script, fix-xmlsec1.txt. +To resolve it: - wget http://www.cloudera.com/documentation/other/shared/fix-xmlsec1.txt -O fix-xmlsec1.py +1. Download the `fix-xmlsec1.txt` script and rename it: -Change permissions as appropriate, for example: + ```bash + wget http://www.cloudera.com/documentation/other/shared/fix-xmlsec1.txt -O fix-xmlsec1.py + ``` - chmod 755 fix-xmlsec1.py +2. Make it executable: -In hue.ini, set xmlsec_binary=/fix-xmlsec1.py. -Run fix-xmlsec1.py. + ```bash + chmod 755 fix-xmlsec1.py + ``` -This script repairs the known issue whereby xmlsec1 is not compiled with RetrievalMethod and cannot find the location of the encrypted key. SAML2 responses would sometimes place EncryptedKey outside of the EncryptedData tree. This script moves EncryptedKey under EncryptedData. +3. In `hue.ini`, set `xmlsec_binary=/fix-xmlsec1.py`. +The script fixes a known issue: `xmlsec1` is not compiled with `RetrievalMethod` and cannot find the encrypted key. SAML2 responses sometimes place `EncryptedKey` outside of the `EncryptedData` tree. The script moves `EncryptedKey` under `EncryptedData`. -### Spnego +### SPNEGO -SPNEGO is an authentication mechanism negotiation protocol. Authentication can be delegated to an authentication server, such as a Kerberos KDC, depending on the mechanism negotiated. +SPNEGO is an authentication mechanism negotiation protocol. Depending on the negotiated mechanism, authentication can be delegated to an authentication server, such as a Kerberos KDC. - desktop.auth.backend.SpnegoDjangoBackend +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.SpnegoDjangoBackend +``` ### PAM -Authenticates users with PAM (pluggable authentication module). The authentication mode depends on the PAM module used. +This backend authenticates users with PAM (pluggable authentication modules). The authentication mode depends on the PAM module you use. - desktop.auth.backend.PamBackend +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.PamBackend +``` -### OAuth Connect +### OAuth -Delegates authentication to a third-party OAuth server. +This backend delegates authentication to a third-party OAuth server. - desktop.auth.backend.OAuthBackend +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.OAuthBackend +``` -### Multiple Authentication Backends +### Multiple authentication backends -For example, to enable Hue to first attempt LDAP directory lookup before falling back to the database-backed user model, we can update the hue.ini configuration file or Hue safety valve in Cloudera Manager with a list containing first the LdapBackend followed by either the ModelBackend or custom AllowFirstUserDjangoBackend (permits first login and relies on user model for all subsequent authentication): +You can list several backends. Hue tries them in order. - [desktop] - [[auth]] - backend=desktop.auth.backend.LdapBackend,desktop.auth.backend.AllowFirstUserDjangoBackend +For example, to try an LDAP lookup first and fall back to the database-backed user model, list `LdapBackend` first and `AllowFirstUserDjangoBackend` second: -This tells Hue to first check against the configured LDAP directory service, and if the username is not found in the directory, then attempt to authenticate the user with the Django user manager. +```ini +[desktop] + [[auth]] + backend=desktop.auth.backend.LdapBackend,desktop.auth.backend.AllowFirstUserDjangoBackend +``` + +Hue first checks the configured LDAP directory. If it does not find the username there, it authenticates the user with the Django user manager. [Read more about it here](http://gethue.com/configuring-hue-multiple-authentication-backends-and-ldap/). ## Security -### Configure a Proxy +### Configure a proxy -We explained how to run Hue with NGINX serving the static files or under Apache. If you use another proxy, you might need to set these options: +If you run Hue behind a proxy other than NGINX or Apache, you might need to set these options: -
-  [desktop]
+```ini
+[desktop]
   # Enable X-Forwarded-Host header if the load balancer requires it.
   use_x_forwarded_host=false
 
   # Support for HTTPS termination at the load-balancer level with SECURE_PROXY_SSL_HEADER.
   secure_proxy_ssl_header=false
-
+``` -### Configuring SSL +### Configure SSL You can configure Hue to serve over HTTPS. -1. Configure Hue to use your private key by adding the following -options to the `hue.ini` configuration file: - - ssl_certificate=/path/to/certificate - ssl_private_key=/path/to/key - -2. Ideally, you would have an appropriate key signed by a Certificate Authority. -If you're just testing, you can create a self-signed key using the `openssl` -command that may be installed on your system: - -Create a key: - - openssl genrsa 1024 > host.key - -Create a self-signed certificate: - - openssl req -new -x509 -nodes -sha1 -key host.key > host.cert +1. Point Hue to your private key and certificate in `hue.ini`: + ```ini + [desktop] + ssl_certificate=/path/to/certificate + ssl_private_key=/path/to/key + ``` -
-Self-signed Certificates and File Uploads +2. Use a key signed by a Certificate Authority. For testing only, you can create a self-signed certificate with `openssl`: -To upload files using the Hue File Browser over HTTPS requires -using a proper SSL Certificate. Self-signed certificates don't -work. -
+ ```bash + # Create a key + openssl genrsa 2048 > host.key -Note: The security vulnerability SWEET32 is also called Birthday attacks against TLS ciphers with 64bit block size and it is assigned CVE-2016-2183. This is due to legacy block ciphers -having block size of 64 bits are vulnerable to a practical collision attack when used in CBC mode. + # Create a self-signed certificate + openssl req -new -x509 -nodes -sha256 -key host.key > host.cert + ``` -DES/3DES are the only ciphers has block size of 64-bit. One way to config Hue not to use them: +> **Note**: To upload files with the File Browser over HTTPS, you need a proper SSL certificate. Self-signed certificates do not work. - [desktop] - ssl_cipher_list=DEFAULT:!DES:!3DES +> **Note**: The SWEET32 vulnerability (CVE-2016-2183) affects legacy block ciphers with a 64-bit block size, used in CBC mode. DES and 3DES are the only ciphers with a 64-bit block size. To stop Hue from using them: +> +> ```ini +> [desktop] +> ssl_cipher_list=DEFAULT:!DES:!3DES +> ``` ### SASL -When getting a bigger result set from Hive/Impala or bigger files like images from HBase, the response requires to increase -the buffer size of SASL lib for thrift sasl communication. +Large result sets from Hive or Impala, or large files such as images from HBase, can require a bigger SASL buffer for Thrift SASL communication. -
-  [desktop]
-  # This property specifies the maximum size of the receive buffer in bytes in thrift sasl communication,
-  # default value is 2097152 (2 MB), which equals to (2 * 1024 * 1024)
+```ini
+[desktop]
+  # Maximum size of the receive buffer in bytes in Thrift SASL communication.
+  # Default is 2097152 (2 MB), which equals 2 * 1024 * 1024.
   sasl_max_buffer=2097152
-
+``` -### Storing passwords in file script +### Store passwords in a script -Hue lets you secure passwords in one consolidated script, or multiple individual scripts. Hue runs each password script at startup and extracts passwords from stdout. +Hue can read passwords from one consolidated script or from several individual scripts. Hue runs each password script at startup and reads the password from its standard output. -Store scripts in a directory that only Hue can read, write, and execute. You can choose password script names but you cannot change hue.ini property names to which you assign those scripts. +Store the scripts in a directory that only the Hue user can read, write, and execute. You can choose the script names, but you cannot change the `hue.ini` property names that you assign them to. -At the command line, create one or more password scripts. For example, create a consolidated script named `my_passwords_script.sh`: +For example, create a consolidated script named `password_script.sh`: - #!/bin/bash +```bash +#!/bin/bash - SERVICE=$1 +SERVICE=$1 - if [[ ${SERVICE} == "ldap_password" ]] - then - echo "your_ldap_password" - fi +if [[ ${SERVICE} == "ssl_password" ]] +then + echo "your_ssl_password" +fi - if [[ ${SERVICE} == "ssl_password" ]] - then - echo "your_ssl_password" - fi +if [[ ${SERVICE} == "bind_password" ]] +then + echo "your_bind_password" +fi - if [[ ${SERVICE} == "bind_password" ]] - then - echo "your_bind_password" - fi +if [[ ${SERVICE} == "db_password" ]] +then + echo "your_database_password" +fi +``` - if [[ ${SERVICE} == "db_password" ]] - then - echo "your_database_password" - fi +Then reference it in `hue.ini`: -Add script properties, for example: +```ini +[desktop] + ssl_password_script="/var/lib/hue/password_script.sh ssl_password" - [desktop] - ldap_username=hueservice - ldap_password_script="/var/lib/hue/password_script.sh ldap_password" - ssl_password_script="/var/lib/hue/password_script.sh ssl_password" - - [[ldap]] + [[ldap]] bind_password_script="/var/lib/hue/password_script.sh bind_password" - [[database]] - db_password_script="/var/lib/hue/password_script.sh db_password" - - -### Idle session timeout - -Hue now offers a new property, idle_session_timeout, that can be configured in the hue.ini file: - - [desktop] - [[auth]] - idle_session_timeout=600 - -When idle_session_timeout is set, users will automatically be logged out after N (e.g. – 600) seconds of inactivity and be prompted to login again: - -[Read more about it here](http://gethue.com/introducing-the-new-login-modal-and-idle-session-timeout/). - -### Login + [[database]] + password_script="/var/lib/hue/password_script.sh db_password" +``` -In the section: +### Login and session settings - [desktop] - [[auth]] - -Users will expire after they have not logged in for 'n' amount of seconds. A negative number means that users will never expire. +These properties are in the `[[auth]]` section: +```ini +[desktop] + [[auth]] + # Users expire after n seconds without logging in. A negative number means users never expire. expires_after=-1 -Apply 'expires_after' to superusers. - + # Apply 'expires_after' to superusers. expire_superusers=true -Users will automatically be logged out after 'n' seconds of inactivity. A negative number means that idle sessions will not be timed out. - + # Log users out after n seconds of inactivity. A negative number means idle sessions never time out. idle_session_timeout=-1 -Force users to change password on first login with desktop.auth.backend.AllowFirstUserDjangoBackend - + # Force users to change their password on first login with desktop.auth.backend.AllowFirstUserDjangoBackend. change_default_password=false -Number of login attempts allowed before a record is created for failed logins - + # Number of login attempts allowed before a record is created for failed logins. login_failure_limit=3 -After number of allowed login attempts are exceeded, do we lock out this IP and optionally user agent? - + # Lock out an IP (and optionally a user agent) after the allowed login attempts are exceeded. login_lock_out_at_failure=false -If set, defines period of inactivity in hours after which failed logins will be forgotten. A value of 0 or None will disable this check. Default: None - + # Period of inactivity, in hours, after which failed logins are forgotten. + # A value of 0 or None disables this check. login_cooloff_time=None -If True, lock out based on an IP address AND a user agent. This means requests from different user agents but from the same IP are treated differently. - + # Lock out based on an IP address AND a user agent. Requests from different user agents + # but from the same IP are then treated differently. login_lock_out_use_user_agent=false -If True, lock out based on IP and user - + # Lock out based on a combination of user and IP. login_lock_out_by_combination_user_and_ip=false +``` -### Auditing +When `idle_session_timeout` is set (for example, `idle_session_timeout=600`), users are logged out after that many seconds of inactivity and are prompted to log in again. -Read more about [Auditing User Administration Operations with Hue and Cloudera Navigator](http://gethue.com/auditing-user-administration-operations-with-hue-and-cloudera-navigator-2/). +[Read more about it here](http://gethue.com/introducing-the-new-login-modal-and-idle-session-timeout/). -### Concurrent User Session Limit +### Concurrent user session limit -If set, limits the number of concurrent user sessions. 1 represents 1 browser session per user. Default: 0 (unlimited sessions per user) +You can limit the number of concurrent sessions per user. A value of `1` allows one browser session per user. The default is `0` (unlimited). - [desktop] - [[session]] +```ini +[desktop] + [[session]] concurrent_user_session_limit=0 +``` [Read more about it here](http://gethue.com/restrict-number-of-concurrent-sessions-per-user/). ## Query execution timeouts -i.e. if the SQL query takes more than 30 seconds, it might timeout without it: +A long SQL query can time out in a proxy with an error like this: - 504 Gateway Time-out 504 Gateway Time-out nginx/1.17.10 +```text +504 Gateway Time-out nginx/1.17.10 +``` -One solution is to execute long blocking or resource intensive operations in tasks outside of the server. This is particularly useful for querying databases other than Hive or Impala that uses the SqlAlchemy interface which is blocking until the query actually finishes. +One solution is to run long, blocking, or resource-intensive operations in tasks outside of the server. This is particularly useful for databases other than Hive or Impala that use the SQLAlchemy interface, which blocks until the query finishes. -Read more in the [Task Server section](/administrator/administration/reference/#task-server) of the Reference Guide. +Read more in the [Task Server section](/administrator/administration/reference/#task-server) of the reference guide. -An alternative is to increase the NGINX timeouts from 30s to more minutes: +Another solution is to increase the NGINX timeouts from 30 seconds to several minutes. For example, with the Kubernetes NGINX ingress: - nginx.ingress.kubernetes.io/proxy-connect-timeout: '900' - nginx.ingress.kubernetes.io/proxy-read-timeout: '900' - nginx.ingress.kubernetes.io/proxy-send-timeout: '900' +```yaml +nginx.ingress.kubernetes.io/proxy-connect-timeout: '900' +nginx.ingress.kubernetes.io/proxy-read-timeout: '900' +nginx.ingress.kubernetes.io/proxy-send-timeout: '900' +``` ## Customization ### Language -The user language is auto-detected and can also be manually overriden as a [per user basis](/user/concept/#changing-the-language). +Hue detects the user language automatically. Users can also override it on a [per-user basis](/user/concept/#changing-the-language). -If text strings are not all translated from English, feel free to improve the `django.po` files of the corresponding language and [recompile them](/developer/development/#internationalization). Those translation mappings are located across the project in [locale](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/locale) directories. +If some text strings are not translated from English, you can improve the `django.po` files of the language and [recompile them](/developer/development/#internationalization). The translation files are in the [locale](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/locale) directories. ### Maps look and feel -The properties we need to tweak are leaflet_tile_layer and leaflet_tile_layer_attribution, that can be configured in the hue.ini file: +Change the `leaflet_tile_layer` and `leaflet_tile_layer_attribution` properties: - [desktop] - leaflet_tile_layer=https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x} - leaflet_tile_layer_attribution='Tiles © Esri — Source: Esri, i-cubed, USDA, USGS, AEX, GeoEye, Getmapping, Aerogrid, IGN, IGP, UPR-EGP, and the GIS User Community' +```ini +[desktop] + leaflet_tile_layer=https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x} + leaflet_tile_layer_attribution='Tiles © Esri — Source: Esri, i-cubed, USDA, USGS, AEX, GeoEye, Getmapping, Aerogrid, IGN, IGP, UPR-EGP, and the GIS User Community' +``` [Read more about it here](http://gethue.com/change-your-maps-look-and-feel/). ### Banner -You can add a custom banner to the Hue Web UI by applying HTML directly to the property, banner_top_html. For example: +Add a custom banner to the Hue web UI by setting HTML in `banner_top_html`: + +```ini +[desktop] + [[custom]] banner_top_html=

My company's custom Hue Web UI banner

+``` + +### Splash screen -### Splash Screen -You can customize a splash screen on the login page by applying HTML directly to the property, login_splash_html. For example: +Add a splash screen to the login page by setting HTML in `login_splash_html`: - [desktop] - [[custom]] +```ini +[desktop] + [[custom]] login_splash_html=WARNING: You are required to have authorization before you proceed. +``` -### Custom Logo +### Custom logo -There is also the possibility to change the logo for further personalization. +Replace the logo with your own SVG code in `logo_svg`. The SVG replaces the default logo in the top bar and on the sign-in screen. Design it to fit in a 160×40 pixel space. - [desktop] - [[custom]] - # SVG code to replace the default Hue logo in the top bar and sign in screen +```ini +[desktop] + [[custom]] # e.g. logo_svg= +``` -You can go crazy and write there any SVG code you want. Please keep in mind your SVG should be designed to fit in a 160×40 pixels space. To have the same ‘hearts logo' you can see above, you can type this code +For example, this code shows a logo made of three hearts: - [desktop] - [[custom]] +```ini +[desktop] + [[custom]] logo_svg=' ' +``` -Read more about it in [Hue with a custom logo](http://gethue.com/hue-with-a-custom-logo/) post. +[Read more about it here](http://gethue.com/hue-with-a-custom-logo/). -## Source Version Control +## Source version control -By default Hue stores the [saved documents](/user/concept/#documents) in its database. This features aims at pointing to any source versioning systems like GitHub, BitBucket... to open and save queries. +By default, Hue stores [saved documents](/user/concept/#documents) in its database. You can point Hue to a source version control system such as GitHub or Bitbucket to open and save queries. -**Note** This feature is experimental and tracked in [HUE-951](https://issues.cloudera.org/browse/HUE-951). +> **Note**: This feature is experimental. - [desktop] +```ini +[desktop] + [[vcs]] + [[[git-read-only]]] + # Base URL to the remote server + remote_url=https://github.com/cloudera/hue/tree/master - [[vcs]] + # Base URL to the version control API + api_url=https://api.github.com - ## [[[git-read-only]]] - ## Base URL to Remote Server - # remote_url=https://github.com/cloudera/hue/tree/master + [[[github]]] + remote_url=https://github.com/cloudera/hue/tree/master + api_url=https://api.github.com - ## Base URL to Version Control API - # api_url=https://api.github.com + # Required to write back to the repository. + # Client ID for the authorized application + client_id= + # Client secret for the authorized application + client_secret= - ## [[[github]]] - ## Base URL to Remote Server - # remote_url=https://github.com/cloudera/hue/tree/master + [[[svn]]] + remote_url=https://github.com/cloudera/hue/tree/master + api_url=https://api.github.com - ## Base URL to Version Control API - # api_url=https://api.github.com + # Required to write back to the repository. + client_id= + client_secret= +``` - # These will be necessary when you want to write back to the repository. - ## Client ID for Authorized Application - # client_id= +## Slack - ## Client Secret for Authorized Application - # client_secret= +Currently in **beta**. - ## [[[svn]] - ## Base URL to Remote Server - # remote_url=https://github.com/cloudera/hue/tree/master +This [integration](/user/concept/#slack) helps users with their SQL queries and lets them collaborate and discuss with other users via Slack. - ## Base URL to Version Control API - # api_url=https://api.github.com +### Architecture - # These will be necessary when you want to write back to the repository. - ## Client ID for Authorized Application - # client_id= +![Share to Slack Architecture](https://cdn.gethue.com/uploads/2021/04/share_to_slack_architecture.png) - ## Client Secret for Authorized Application - # client_secret= +One flow shares query and gist links. The bot logic runs on top of the Hue server. It listens to the events that the Slack Hue app posts to an endpoint. It processes them, for example by generating a rich preview that unfurls the links shared in Slack channels. It then sends the responses back to Slack with the Slack API. -## Slack -Currently in **Beta** +The Smart Query Assistance block on top of the Hue server drives the other flow, SQL assistance. It is a work in progress. It is meant to answer users who ask how to find certain data tables or query them. -This [integration](/user/concept/#slack) with Hue helps users by assisting them with their SQL queries and have better collaboration/discussion with other users via Slack. +### Easier Slack app installation -### Architectural View -![Share to Slack Architecture](https://cdn.gethue.com/uploads/2021/04/share_to_slack_architecture.png) +The **Slack workspace admin** follows these steps to set up a Hue Slack app. You create the app only once. Other Hue users can then [interact](/user/concept/#slack) with the app in Slack channels. + +1. Go to https://gethue.com/ and, in the last section, enter your **Hue instance hostname** (for example, `hue.gethue.com:8000`). +2. Choose the Slack workspace where you want to install the app and click **Next**. +3. Review the configuration and click **Create**. +4. Install the app in the workspace and connect it to Hue with the [last step of the manual installation](#update-your-hueini-file). -One of the flows for sharing query/gist links, the main bot logic lies on top of the Hue server listening to the events posted by the Hue App from Slack to an endpoint, processing those events such as generating a rich preview to unfurl for the links shared in the Slack channels and using Slack API methods for sending these responses back to Slack. +![Easier Installation GIF](https://cdn.gethue.com/uploads/2021/06/easier_installation_slack.gif) -The 'Smart Query Assistance' block on top of Hue server which drives the other flow for SQL Assistance is currently a work in progress for replying to users asking questions on how to find certain data tables or to query them. +### Manual Slack app installation -### Improved Slack App Installation -Steps to be followed by the **Slack workspace admin** to set up their own Hue Slack app. The app needs to be created only once, other Hue users can simply [interact](/user/concept/#slack) with the App in the Slack channels! +#### App manifest (YAML) -- Go to https://gethue.com/ and in the last section, give your **Hue instance hostname** _(e.g. hue.gethue.com:8000)_. -- Choose the Slack workspace where you want to install the app and click **Next**. -- You can review the configuration and simply click **Create**. -- Once the app is created, install it in the workspace and plug-it with Hue by following this [last step](/administrator/configuration/server/#updating-your-hueini-config-file). +The latest manifest file is checked in [here](https://github.com/cloudera/hue/blob/master/tools/slack/manifest.yml). -![Easier Installation GIF](https://cdn.gethue.com/uploads/2021/06/easier_installation_slack.gif) +#### Changes needed in the manifest -### Manual Slack App Installation +Replace the two `demo.gethue.com` values with **your Hue instance hostname**: -#### App Manifest (YAML) -The latest version of the manifest file is checked-in [here](https://github.com/cloudera/hue/blob/master/tools/slack/manifest.yml). +* Under `unfurl_domains`. +* Under `event_subscriptions`, in `request_url`: `https:///desktop/slack/events/`. -#### Changes needed in the YAML Manifest -Update the two _demo.gethue.com_ with **your Hue instance hostname:** -- Under **unfurl_domains** -- Under **event_subscriptions**, in **request_url** `https:///desktop/slack/events/` +#### Create your app -#### Creating your app 1. Go to https://api.slack.com/apps and click **Create New App**. -2. Choose **From an app manifest** option and workspace where you want to install the app and click **Next**. -3. Choose **YAML** and paste the Manifest code (make sure you do the necessary changes mentioned above) and click **Next**. -4. Read the review summary and if everything’s correct, click **Create**. -5. Once the app is created, install it in the workspace. - -#### Updating your hue.ini config file -6. Go to the **OAuth & Permissions** page, copy the **Bot User OAuth Token** and update **slack_bot_user_token** (e.g. _xoxb-xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxx_). -7. Similarly, go to the **Basic Information** page, copy the **Verification Token** and update **slack_verification_token**. -8. Paste this in your hue.ini file under `[desktop]` section. - - [[slack]] - is_enabled=true - slack_verification_token= - slack_bot_user_token= - -Now add the Slack app in your desired channels and send some [query/gist links](/user/concept/#slack). +2. Choose **From an app manifest**, select the workspace where you want to install the app, and click **Next**. +3. Choose **YAML**, paste the manifest (with the changes above), and click **Next**. +4. Read the review summary. If everything is correct, click **Create**. +5. Install the app in the workspace. + +#### Update your hue.ini file + +1. Go to the **OAuth & Permissions** page and copy the **Bot User OAuth Token** (for example, `xoxb-xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxx`). +2. Go to the **Basic Information** page and copy the **Verification Token**. +3. Add both to the `[desktop]` section of your `hue.ini` file: + +```ini +[desktop] + [[slack]] + is_enabled=true + slack_verification_token= + slack_bot_user_token= +``` + +Now add the Slack app to your channels and send some [query or gist links](/user/concept/#slack). diff --git a/docs/docs-site/content/administrator/installation/_index.md b/docs/docs-site/content/administrator/installation/_index.md index 6aa238cecea..d245fcb8374 100644 --- a/docs/docs-site/content/administrator/installation/_index.md +++ b/docs/docs-site/content/administrator/installation/_index.md @@ -5,22 +5,21 @@ draft: false weight: 1 --- -## Quick Start +## Quick start -If you just want to: - -* Start Hue, follow the [Container](/administrator/installation/cloud/) section -* Test some small code changes, see the [Development Quick Start](https://github.com/cloudera/hue#getting-started) +To run Hue: +* In a container, follow the [Container](/administrator/installation/cloud/) section. +* From source, install the [dependencies](/administrator/installation/dependencies/) and follow the [Install](/administrator/installation/install/) section. +* To test small code changes, see the [development quick start](/developer/development/#quick-start). ## Installation -Continue to the [next sections](/administrator/installation/dependencies/) for detailed instructions on how to install & run the service. +Continue to the [next sections](/administrator/installation/dependencies/) for detailed instructions on how to install and run the service. + +## Third-party distributions -You can also find distributions of Hue via these companies: +Other companies distribute Hue. Their versions can differ from the open-source release: -* [Cloudera Data Platform - Hue](https://www.cloudera.com/products/cloudera-data-platform.html) -* Amazon AWS EMR - Hue -* [Google Cloud Dataproc - Hue](https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/tree/master/hue) -* [Azure HDInsight - Hue](https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-hue-linux) -* Open Data Hub - Data Catalog +* [Google Cloud Dataproc](https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/tree/master/hue) +* [Azure HDInsight](https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-hue-linux) diff --git a/docs/docs-site/content/administrator/installation/cloud.md b/docs/docs-site/content/administrator/installation/cloud.md index 9e0d1218a0d..897563e936a 100644 --- a/docs/docs-site/content/administrator/installation/cloud.md +++ b/docs/docs-site/content/administrator/installation/cloud.md @@ -7,181 +7,207 @@ weight: 3 ## Process -After the installation, you can start Hue on your Hue Server by doing: +After you [install Hue](/administrator/installation/install/), start it from the installation directory: - build/env/bin/supervisor +```bash +build/env/bin/supervisor +``` -This will start several subprocesses, corresponding to the different Hue -components. Your Hue installation is now running. +The supervisor starts several subprocesses, one for each Hue component. -After installation, you can use Hue by navigating to `http://myserver:8888/`. +Open `http://myserver:8888/` to use Hue. -Next step is to [configure](/administrator/configuration/) the API server and point to external services. +Next, [configure](/administrator/configuration/) the API server and point it to external services. ## Docker To start a Hue instance: - docker run -it -p 8888:8888 gethue/hue:latest +```bash +docker run -it -p 8888:8888 gethue/hue:latest +``` -In addition to providing a ready to use image, the [Docker Guide](https://github.com/cloudera/hue/tree/testing/tools/docker) shows how to build it. +The [Docker guide](https://github.com/cloudera/hue/tree/master/tools/docker/hue) shows how to build the image, start the containers, and configure them. -It then details how to start the containers and parameterize them. +> **Note:** The image uses an embedded SQLite database by default. Connect Hue to [MySQL or PostgreSQL](/administrator/administration/database/) before real use. ## Kubernetes To start a full Hue service: - helm repo add gethue https://helm.gethue.com - helm repo update - helm install hue gethue/hue +```bash +helm repo add gethue https://helm.gethue.com +helm repo update +helm install hue gethue/hue +``` -The [Kubernetes Guide](https://github.com/cloudera/hue/tree/testing/tools/kubernetes) shows how to run the services via Helm or native YAML configs. +The [Kubernetes guide](https://github.com/cloudera/hue/tree/master/tools/kubernetes) and the [Hue Helm chart](https://github.com/cloudera/hue/tree/master/tools/kubernetes/helm/hue) show how to run the services with Helm. ### Logs -Fluentd is a log management system that is heavily used in the Kubernetes world. Here we will leverage the Microk8s distribution that bundles it. - - First we install the fluentd add-on: - - microk8s.enable fluentd - -And see that the Elastic Search, Fluentd and Kibana UI are running: - - kubectl get pods -A - NAMESPACE NAME READY STATUS RESTARTS AGE - container-registry registry-577986746b-btjdz 1/1 Running 299 96d - default hue-rp2qf 1/1 Running 69 32d - default nginx-hue-85df47ddd7-bbmwk 1/1 Running 129 48d - default postgres-hue-6cpsz 1/1 Running 72 32d - kube-system coredns-5874dcd95f-cnkfl 1/1 Running 239 84d - kube-system elasticsearch-logging-0 1/1 Running 117 47d - kube-system fluentd-es-v2.2.0-pp7qb 1/1 Running 401 33d - kube-system hostpath-provisioner-6d744c4f7c-9dgnv 1/1 Running 123 47d - kube-system kibana-logging-df8d4c8fd-kms74 1/1 Running 169 57d - kube-system kube-dns-6bfbdd666c-gzbh9 3/3 Running 369 47d - kube-system metrics-server-v0.2.1-57dfcb796b-9v7dn 2/2 Running 832 114d - kube-system tiller-deploy-765dcb8745-zlm6t 1/1 Running 382 106d - -Microk8s automatically ships the logs of each container. If you are curious, here is one way to see its configuration: - - kubectl edit configmaps fluentd-es-config-v0.1.5 -n kube-system - - containers.input.conf: |2- - - - @id fluentd-containers.log - @type tail - path /var/log/containers/*.log - pos_file /var/log/es-containers.log.pos - tag raw.kubernetes.* - read_from_head true - - @type multi_format - - format json - time_key time - time_format %Y-%m-%dT%H:%M:%S.%NZ - - - format /^(? - - - - - @id raw.kubernetes - @type detect_exceptions - remove_tag_prefix raw - message log - stream stream - multiline_flush_interval 5 - max_bytes 500000 - max_lines 1000 - - -Fluentd injects some metadata to each log line. It contains information about Kubernetes properties like container and image names. Now let’s open the Kibana UI and look at the “hue” container logs: - - - https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now%2Fd,mode:quick,to:now%2Fd))&_a=(columns:!(log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b51668f0-a9aa-11e9-afc4-53db22981ed0,key:kubernetes.container_name,negate:!f,params:(query:hue,type:phrase),type:phrase,value:hue),query:(match:(kubernetes.container_name:(query:hue,type:phrase))))),index:b51668f0-a9aa-11e9-afc4-53db22981ed0,interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc)) - -The credentials can be seen by looking at the bottom of the command: - - microk8s.config - -Then select the indexes with the “logs*” patterns and use “@timestamp” as the time field. And then open above link or the Discover tab to see all the Hue container logs in real time. - -It becomes then easy to look at what the users are doing, which errors they are facing… For example, here is one way to look at how many query have been executed: - -Next step: in addition to Elastic Search, the logs can also be automatically stored into additional outputs like HDFS, S3 or sent to Kafka. +Fluentd is a log management system that is widely used in Kubernetes. This section uses the MicroK8s distribution, which bundles it. + +Install the Fluentd add-on: + +```bash +microk8s.enable fluentd +``` + +Check that Elasticsearch, Fluentd, and Kibana are running: + +```bash +kubectl get pods -A +``` + +Example output: + +```text +NAMESPACE NAME READY STATUS RESTARTS AGE +container-registry registry-577986746b-btjdz 1/1 Running 299 96d +default hue-rp2qf 1/1 Running 69 32d +default nginx-hue-85df47ddd7-bbmwk 1/1 Running 129 48d +default postgres-hue-6cpsz 1/1 Running 72 32d +kube-system coredns-5874dcd95f-cnkfl 1/1 Running 239 84d +kube-system elasticsearch-logging-0 1/1 Running 117 47d +kube-system fluentd-es-v2.2.0-pp7qb 1/1 Running 401 33d +kube-system hostpath-provisioner-6d744c4f7c-9dgnv 1/1 Running 123 47d +kube-system kibana-logging-df8d4c8fd-kms74 1/1 Running 169 57d +kube-system kube-dns-6bfbdd666c-gzbh9 3/3 Running 369 47d +kube-system metrics-server-v0.2.1-57dfcb796b-9v7dn 2/2 Running 832 114d +kube-system tiller-deploy-765dcb8745-zlm6t 1/1 Running 382 106d +``` + +MicroK8s automatically ships the logs of each container. If you are curious, here is one way to see its configuration: + +```bash +kubectl edit configmaps fluentd-es-config-v0.1.5 -n kube-system +``` + +The `containers.input.conf` entry looks like this: + +```text +containers.input.conf: |2- + + + @id fluentd-containers.log + @type tail + path /var/log/containers/*.log + pos_file /var/log/es-containers.log.pos + tag raw.kubernetes.* + read_from_head true + + @type multi_format + + format json + time_key time + time_format %Y-%m-%dT%H:%M:%S.%NZ + + + format /^(? + + + + + @id raw.kubernetes + @type detect_exceptions + remove_tag_prefix raw + message log + stream stream + multiline_flush_interval 5 + max_bytes 500000 + max_lines 1000 + +``` + +Fluentd injects some metadata to each log line. It contains information about Kubernetes properties like container and image names. Open the Kibana UI and look at the `hue` container logs: + + +```text +https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now%2Fd,mode:quick,to:now%2Fd))&_a=(columns:!(log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b51668f0-a9aa-11e9-afc4-53db22981ed0,key:kubernetes.container_name,negate:!f,params:(query:hue,type:phrase),type:phrase,value:hue),query:(match:(kubernetes.container_name:(query:hue,type:phrase))))),index:b51668f0-a9aa-11e9-afc4-53db22981ed0,interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc)) +``` + +The credentials are at the bottom of the output of this command: + +```bash +microk8s.config +``` + +Select the indexes with the `logs*` pattern and use `@timestamp` as the time field. Then open the link above or the Discover tab to see the Hue container logs in real time. + +You can then check what users are doing and which errors they face. Fluentd can also send the logs to other outputs like HDFS, S3, or Kafka. ### Metrics -Hue [metrics](/administrator/administration/operations#metrics) are useful for checking the load (how many users), slowness (average or percentile times taken by requests)... Those have been available via the /metrics page, but here is how to collect and aggregate this information in Kubernetes. +Hue [metrics](/administrator/administration/operations/#metrics) show the load (number of users) and slowness (average or percentile request times). This section shows how to collect them in Kubernetes with Prometheus. -Prometheus is the metric collecting system heavily used in the Kubernetes world. +This section assumes the Prometheus operator is running and powers the Prometheus pods in the `monitoring` namespace: -First we suppose we have the Prometheus operator running, which powers the Prometheus pods in the monitoring namespace: +```bash +kubectl get pods -n monitoring +``` - kubectl get pods -n monitoring - NAME READY STATUS RESTARTS AGE - alertmanager-main-0 2/2 Running 268 48d - grafana-7789c44cc7-7c4pb 1/1 Running 125 48d - kube-state-metrics-78c549dd89-kwmwg 4/4 Running 512 48d - node-exporter-zlg4s 2/2 Running 259 48d - prometheus-adapter-644b448b48-7t8rt 1/1 Running 131 48d - prometheus-k8s-0 3/3 Running 364 47d - prometheus-operator-7695b59fb8-k2qm2 1/1 Running 130 48d +Example output: -To tell Prometheus how to get the metrics, we use a ServiceMonitor. Those metrics are available on the /metrics page of Hue via the Django Prometheus module. Note that to expose this URL, Hue needs to have this property turned on: +```text +NAME READY STATUS RESTARTS AGE +alertmanager-main-0 2/2 Running 268 48d +grafana-7789c44cc7-7c4pb 1/1 Running 125 48d +kube-state-metrics-78c549dd89-kwmwg 4/4 Running 512 48d +node-exporter-zlg4s 2/2 Running 259 48d +prometheus-adapter-644b448b48-7t8rt 1/1 Running 131 48d +prometheus-k8s-0 3/3 Running 364 47d +prometheus-operator-7695b59fb8-k2qm2 1/1 Running 130 48d +``` - [desktop] - enable_prometheus=true +Use a ServiceMonitor to tell Prometheus where to get the metrics. Hue exposes them on the `/metrics` page with the Django Prometheus module. To expose this URL, turn on this property in the `hue.ini`: -Then we can check that Prometheus is scraping properly Hue: http://prometheus:9090/targets and get a series of metrics to understand how the Hues are behaving: http://prometheus:9090/graph. +```ini +[desktop] +enable_prometheus=true +``` -This was a very basic introduction to metrics of Hue in the Kubernetes ecosystem. In a following next step, we should describe which metrics are particularly useful and how to setup default dashboards and alerts. +Then check that Prometheus scrapes Hue at `http://prometheus:9090/targets`. Browse the metrics at `http://prometheus:9090/graph`. ### Tracing -Hue is getting easy to run with its Docker container and Kubernetes Helm package. Recent blog posts describe how to get access to logs and metrics. Even in a non distributed world it can get noisy to know how much time is being spent where in each user request. - -Consequently, in the context of a Data Analyst, knowing why a certain query is slow can become problematic. On top of that, adding multiple tenants and users, and more than 20 external APIs and the fog about fine grain performances appears and its becomes extremely manual and time consuming to troubleshoot. +Tracing shows how much time each part of a user request takes. This helps you find out why a query is slow. -In order to help get clarity on where exactly each request time is being spent, Hue started to implement the Opentracing API. Jaeger was selected as the implementation for its ease of use and close support with Kubernetes. Here we will also leverage the Microk8s distribution that bundles it. +Hue implements the OpenTracing API and uses Jaeger as the tracer. #### Setup -Hue now ships with the open tracing integration, and details about the current state of this feature are in the Tracing design document. To turn it on, in the hue.ini: - [desktop] - [[tracing]] - ## If tracing is enabled. - enabled=true +To turn on tracing, edit the `hue.ini`: - ## Trace all the requests instead of a few specific ones like the SQL Editor. Much noisier. - trace_all=false +```ini +[desktop] +[[tracing]] +## If tracing is enabled. +enabled=true -On the Jaerger side, as explained in the quick start, it is simple to run it on the same host as Hue with this container: +## Trace all the requests instead of a few specific ones like the SQL Editor. Much noisier. +trace_all=false +``` - docker run -d --name jaeger \ - -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ - -p 5775:5775/udp \ - -p 6831:6831/udp \ - -p 6832:6832/udp \ - -p 5778:5778 \ - -p 16686:16686 \ - -p 14268:14268 \ - -p 9411:9411 \ - jaegertracing/all-in-one:1.14 +Run Jaeger on the same host as Hue with this container: -And that’s it! Jaeger should show up at this page http://localhost:16686. +```bash +docker run -d --name jaeger \ + -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ + -p 5775:5775/udp \ + -p 6831:6831/udp \ + -p 6832:6832/udp \ + -p 5778:5778 \ + -p 16686:16686 \ + -p 14268:14268 \ + -p 9411:9411 \ + jaegertracing/all-in-one:1.14 +``` -#### Tracing queries +The Jaeger UI is then available at `http://localhost:16686`. -In the SQL Editor of Hue, execute a series of queries. In the Jaeger UI, if you then select the hue-api service, each external call to the queried datawarehouse (e.g. execute_statement, fetch_status, fetch_result… to MySql, Apache Impala…) are being traced. Below we can see 5 query executions that went pretty fast. - -Fine grain filtering at the user or query level operation is possible. For example, to lookup all the submit query calls of the user ‘romain’, select ‘notebook-execute’ as the Operation, and tag filter via user-id=”romain”: +#### Tracing queries -In the next iteration, more calls and tags (e.g. filter all traces by SQL session XXX) will be supported and a closer integration with the database engine would even propagate the trace id across all the system. +Run some queries in the SQL Editor. In the Jaeger UI, select the `hue-api` service. Each external call to the queried data warehouse (for example `execute_statement`, `fetch_status`, and `fetch_result`) is traced. diff --git a/docs/docs-site/content/administrator/installation/dependencies/_index.md b/docs/docs-site/content/administrator/installation/dependencies/_index.md index 002d00b733c..b64f8ca9278 100644 --- a/docs/docs-site/content/administrator/installation/dependencies/_index.md +++ b/docs/docs-site/content/administrator/installation/dependencies/_index.md @@ -5,331 +5,226 @@ draft: false weight: -1 --- -You will need Python, some OS packages, a Database and Node.js. +You need Python, some OS packages, a database, and Node.js to build and run Hue. ## Python -Hue employs some Python modules which use native code and requires certain development libraries be installed on your system. To install from the tarball, you'll need these library development packages and tools installed on your system: +Hue uses Python modules that compile native code, so you need the Python development libraries and build tools on your system. See [OS packages](#os-packages). -Versions supported: -* Python 3.11, **3.11.18 or later** recommended (see the note below) -``` -# If you are using Python 3.11, set PYTHON_VER before the build, like -export PYTHON_VER=python3.11 +Supported versions: -# Export ROOT which should point to your Hue directory -export ROOT= -``` +* Python 3.11 only. Python 3.11.18 or later is recommended (see the security note below). ->**Security note (eMudhra CA incident):** the compromised eMudhra (emSign) CA was removed from CA bundles such as `certifi`. The Hue build pins `certifi` to `2026.10.23`, installs `pip` 26.2.2 (older `pip` versions keep trusting the compromised CA through their vendored `certifi`) and uses `virtualenv` 21.14.13 on the build host. Python 3.11.18 additionally rejects the compromised intermediate CAs on its own. Rebuild container images without layer caching (`tools/container/build.sh` does this by default) so they pick up the latest system certificates. +Before you build, set the Python version: -## Database - -Hue is being ran the most with [MySQL InnoDB, PostgreSQL or Oracle](https://docs.cloudera.com/documentation/enterprise/latest/topics/hue_dbs_0.html). - -### MySQL / MariaDB - -The client lib [MySQL-python](https://github.com/cloudera/hue/tree/master/desktop/core/ext-py/MySQL-python-1.2.5) is already included so it should work out of the box. - -**Note** +```bash +export PYTHON_VER=python3.11 +``` -With the recent OSs like Ubuntu 20.04 or using Python 3, the MySQL-python lib won't compile properly and will produce an error similar to: +> **Security note (eMudhra CA incident):** the compromised eMudhra (emSign) CA was removed from CA bundles such as `certifi`. The Hue build pins `certifi` to `2026.10.23`, installs `pip` 26.2.2 (older `pip` versions keep trusting the compromised CA through their vendored `certifi`) and uses `virtualenv` 21.14.13 on the build host. Python 3.11.18 additionally rejects the compromised intermediate CAs on its own. Rebuild container images without layer caching (`tools/container/build.sh` does this by default) so they pick up the latest system certificates. - _mysql.c:44:10: fatal error: my_config.h: No such file or directory - 44 | #include "my_config.h" - | ^~~~~~~~~~~~~ +## Database -The lib folder needs to be swapped with https://pypi.org/project/mysqlclient/. Unfortunately for licensing reason (GPL vs Apache) this can't be officially done in the Hue repository. +Hue stores its data (users, query history, sharing permissions) in a database. It runs best with [MySQL InnoDB, MariaDB, PostgreSQL, or Oracle](/administrator/administration/database/). The embedded SQLite database is for testing only. -Remove and add the compatible MySql lib with these two commands: +Install the Python client library of your database in the Hue virtual environment, after `make apps` or `make install`. - git cherry-pick 7a9100d4a7f38eaef7bd4bd7c715ac1f24a969a8 - git cherry-pick e67c1105b85b815346758ef1b9cd714dd91d7ea3 +### MySQL / MariaDB - git clean -fdx +Install the OS client headers (`libmysqlclient-dev` on Ubuntu), then install `mysqlclient`: - make apps +```bash +./build/env/bin/pip install mysqlclient +``` -Read [HUE-9390](https://issues.cloudera.org/browse/HUE-9390) for more details. +The Hue Docker image uses `mysqlclient==2.2.7`. Use the `mysql` engine for both MySQL and MariaDB. ### PostgreSQL -You would need the OS `python-psycopg2` or Python [psycopg2](https://pypi.org/project/psycopg2/). - -### Oracle - -How to install the client lib. - -#### Version 11 - -Download both instantclient-basic and instantclient-sdk of the same version (11.2.0.4.0 for this example) and on your ~/.bash_profile, add - - export ORACLE_HOME=/usr/local/share/oracle - export VERSION=11.2.0.4.0 - export ARCH=x86_64 - export DYLD_LIBRARY_PATH=$ORACLE_HOME - export LD_LIBRARY_PATH=$ORACLE_HOME - -and then - - source ~/.bash_profile - sudo mkdir -p $ORACLE_HOME - sudo chmod 775 $ORACLE_HOME - -then unzip the content of both downloaded zip files into the newly created $ORACLE_HOME in a way that the 'sdk' folder is at the same level with the other files and then - - ln -s libclntsh.dylib.11.1 libclntsh.dylib - ln -s libocci.dylib.11.1 libocci.dylib - -and finally +Install `psycopg2-binary`: - cd sdk - unzip ottclasses.zip - -#### Version 12 - -Hue comes with an older Oracle client cx_Oracle-5.2.1 Python module so it will fail. We need to do the above client install and then upgrade Hue's client module to at least cx_Oracle-5.3. - -Make sure you have the `python-dev` package dependencies, ensure that ORACLE_HOME and LB_LIBRARY_PATH are properly set so that pip knows which version to install, then: - - echo $ORACLE_HOME $LD_LIBRARY_PATH - - ./build/env/bin/pip install cx_Oracle - -Tip: You can also wget the proper cx_Oracle file yourself: https://pypi.python.org/pypi/cx_Oracle/. - -Tip: Going to where is the Oracle client, e.g. /usr/local/share/oracle then creating a symlink similar to below could even trick the cx_Oracle-5.2.1 module to work with 12.2 without doing the pip upgrade: +```bash +./build/env/bin/pip install psycopg2-binary +``` - ln -s libclntsh.so.12.2 libclntsh.so.11.1 +### Oracle -There is more details on this [Apply Temporary Workaround for Oracle 12 Client](https://docs.cloudera.com/documentation/enterprise/latest/topics/hue_dbs_oracle_pkg.html#concept_qx3_hfw_4z). +Hue ships the `cx_Oracle` 8.3.0 Python module. It needs the Oracle Instant Client libraries on the host. Download the Basic and SDK packages of the same version from [Oracle Instant Client](https://www.oracle.com/database/technologies/instant-client/downloads.html), unzip them into one directory, and make the directory available to the loader: +```bash +export ORACLE_HOME=/usr/local/share/oracle +export LD_LIBRARY_PATH=$ORACLE_HOME +``` -#### Mac +On macOS, use `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. -* [Oracle Instant Client](https://www.oracle.com/database/technologies/instant-client/downloads.html) +Hue sets `options={"threaded": true}` for Oracle by default. If you override `options` in the `[[database]]` section of the `hue.ini`, keep `threaded` set to `true` to avoid crashes. See [Database](/administrator/administration/database/). -## OS Packages +## OS packages ### Ubuntu - sudo apt-get install git ant gcc g++ libffi-dev libkrb5-dev libmysqlclient-dev libsasl2-dev libsasl2-modules-gssapi-mit libsqlite3-dev libssl-dev libxml2-dev libxslt-dev make maven libldap2-dev python-dev python-setuptools libgmp3-dev - -With Python 3, also: - - sudo apt-get install python3.11-dev python3-distutils +```bash +sudo apt-get install git gcc g++ make pkg-config libffi-dev libkrb5-dev krb5-config libsasl2-dev libsasl2-modules-gssapi-mit libsqlite3-dev libssl-dev libxml2-dev libxslt-dev libldap2-dev libsnappy-dev libmysqlclient-dev python3.11 python3.11-dev python3.11-venv python3.11-distutils +``` +On Ubuntu 22.04, `python3.11` is in the `universe` repository. ### CentOS/RHEL - sudo yum install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel - -* mvn (from [``apache-maven``](https://gist.github.com/sebsto/19b99f1fa1f32cae5d00) package or maven3 tarball) -* libtidy (for unit tests only) -* openssl-devel (for version 7+) +```bash +sudo yum install cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python3.11-devel sqlite-devel gmp-devel openssl-devel +``` ### SLES 12 - zypper in ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap2-devel python-devel sqlite-devel gmp-devel +```bash +zypper in cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap2-devel python311-devel sqlite-devel gmp-devel +``` + +> **Note:** If the Kerberos extension fails to build with `krb5-config: command not found`, install `krb5-devel` and add the directory of `krb5-config` to your `PATH`: -Tip: if you run into building kerberos extension issue and see message `krb5-config: command not found`, install and append it to environment variable PATH. +```bash +rpm -ql krb5-devel | grep krb5-config ## find the path of krb5-config +export PATH=$PATH:/usr/lib/mit/bin +``` - rpm -ql krb5-devel | grep krb5-config ## find the path of krb5-config - export PATH=$PATH:/usr/lib/mit/bin +### macOS -### MacOS +Prerequisites: * Xcode command line tools * [Homebrew](https://brew.sh) #### Apple Silicon (M1, M2, M3) -This is a verified step-by-step guide on how to get Hue up and running on a fresh installation of macOS, tested on Apple Silicon (M1, M2, M3). - -1. Clone the Hue repo +This step-by-step guide was tested on a fresh macOS installation on Apple Silicon. - `git clone https://github.com/cloudera/hue.git` +1. Clone the Hue repository: -2. Install Homebrew (if not already installed) - - `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"` + ```bash + git clone https://github.com/cloudera/hue.git + cd hue + ``` -3. Install Postgres (unless you prefer something else) +2. Install Homebrew if it is not already installed: - `brew install postgresql@14` + ```bash + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + ``` -4. Start Postgres +3. Install and start PostgreSQL (or use another database): - `brew services start postgresql@14` + ```bash + brew install postgresql@14 + brew services start postgresql@14 + ``` -5. Create the Hue database and set permissions +4. Create the Hue database and user: - ``` - # Create the DB + ```bash createdb hue_d --lc-collate='en_US.UTF-8' -T "template0" - - # Connect to the dbms shell psql -s hue_d - - # Run the following lines one by one + ``` + + In the `psql` shell, run these statements one by one: + + ```sql create user hue_u with password 'huepassword'; grant all privileges on database hue_d to hue_u; \q ``` -6. Install gmp, openssl, and libffi (Note: some might be installed from Postgres) +5. Install `gmp`, `openssl`, and `libffi`. Some of them might already be installed with PostgreSQL: - `brew install gmp openssl libffi` - -7. Install Python, tested with the **macOS 64-bit (universal2) installer 3.11.x** direct download from python.org -`https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg` + ```bash + brew install gmp openssl libffi + ``` - - For Apple Silicon (M1, M2, M3), Python 3.11 is recommended as of June 2025. You can install Python 3.11 either via Homebrew or by direct download from python.org—both options will work, but the direct download is preferred and tested. +6. Install Python 3.11, either with Homebrew or with the macOS installer from [python.org](https://www.python.org/downloads/macos/). The direct download from python.org is the tested option. ->**Note:** Hue requires Python 3.11. Set `export PYTHON_VER=python3.11` before building and use `./build/env/bin/hue` for Hue commands. python.org only publishes macOS installers for Python 3.11 up to 3.11.9; to get 3.11.18 or later, install Python 3.11 via Homebrew or build it from source. The Hue build upgrades `pip` and pins `certifi` either way. + > **Note:** Hue requires Python 3.11. python.org only publishes macOS installers for Python 3.11 up to 3.11.9. To get 3.11.18 or later, install Python 3.11 with Homebrew or build it from source. The Hue build upgrades `pip` and pins `certifi` either way. -8. Install node and npm (unless already present) using nvm or brew +7. Install Node.js 22.13 or later with `nvm` or Homebrew: - `brew install node` + ```bash + brew install node + ``` -9. In the Hue directory run: +8. In the Hue directory, set the build variables and build: - ``` - export PYTHON_VER=python3.11 - export SKIP_PYTHONDEV_CHECK=true - export CFLAGS="-I$(xcrun --show-sdk-path)/usr/include/sasl" - export LDFLAGS="-L/usr/local/opt/libffi/lib -L/usr/local/opt/openssl@1.1/lib" - export CPPFLAGS="-L/usr/local/opt/libffi/lib/include -L/usr/local/opt/openssl@1.1/lib/include" - ``` + ```bash + export PYTHON_VER=python3.11 + export SKIP_PYTHONDEV_CHECK=true + export CFLAGS="-I$(xcrun --show-sdk-path)/usr/include/sasl" + export LDFLAGS="-L/usr/local/opt/libffi/lib -L/usr/local/opt/openssl@1.1/lib" + export CPPFLAGS="-L/usr/local/opt/libffi/lib/include -L/usr/local/opt/openssl@1.1/lib/include" + make apps + ``` - Also export `ROOT` which should point to your Hue directory: - ``` - export ROOT= - ``` - - Then run: - ``` - make apps - ``` +9. Create the configuration file and set the database in the `[[database]]` section under `[desktop]` of `desktop/conf/pseudo-distributed.ini`: -10. Configure Hue to your liking, edit + ```bash + cp desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini + ``` - `desktop/conf/pseudo-distributed.ini` + ```ini + [[database]] + engine=postgresql_psycopg2 + host=localhost + port=5432 + user=hue_u + password=huepassword + name=hue_d + ``` - According to step 5 above for Postgres, set under `[desktop]`: +10. Install the PostgreSQL client library in the Hue virtual environment: + ```bash + ./build/env/bin/pip install psycopg2-binary ``` - [[database]] - engine=postgresql_psycopg2 - host=localhost - port=5432 - user=hue_u - password=huepassword - name=hue_d - ``` - -11. If you use Postgres as configured in the previous step you'll need to add the psycopg2 lib manually - `./build/env/bin/pip install psycopg2-binary` +11. Apply the database migrations: -12. Apply migrations (if any) - - ``` - ./build/env/bin/hue makemigrations + ```bash ./build/env/bin/hue migrate ``` -13. Start Hue - - `./build/env/bin/hue runserver` - - -#### macOS 10.14, 10.15 - -Install Dependencies via Homebrew - - brew install mysql@5.7 maven gmp openssl libffi - -Install Xcode command line tools - - sudo xcode-select --install +12. Start Hue: -Fix openssl errors (required for MacOS 10.11+) - - export LDFLAGS=-L/usr/local/opt/openssl/lib && export CPPFLAGS=-I/usr/local/opt/openssl/include - -If `runserver` stops abruptly with a `"zsh: abort"` message - - export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/opt/openssl/lib - -If `runserver` stops abruptly with a `loading libcrypto in an unsafe way` message - - sudo mkdir /usr/local/lib - # depends on your openssl version, link the following files to /usr/local/lib - sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.2/lib/libcrypto.3.dylib /usr/local/lib/libcrypto.dylib - sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.2/lib/libssl.3.dylib /usr/local/lib/libssl.dylib - -If you are getting `Could not find Python.h` message - - export SKIP_PYTHONDEV_CHECK=true - -On macOS 10.15+, install an older version of openssl - - brew uninstall --ignore-dependencies openssl && brew install https://github.com/tebelorg/Tump/releases/download/v1.0.0/openssl.rb - -Fix the possible missing Python headers message by installing the MacOS SDK headers - -On macOS 10.14.x - - open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg + ```bash + ./build/env/bin/hue runserver + ``` -On macOS 10.15.x + Open [http://localhost:8000](http://localhost:8000). - sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/ +If you get a `Could not find Python.h` message, make sure `SKIP_PYTHONDEV_CHECK=true` is exported. -## NodeJs +## Node.js Node.js 22.13 or later is required. -e.g. how to install on Ubuntu: +For example, to install it on Ubuntu: - curl -sL https://deb.nodesource.com/setup_22.x | sudo bash - - sudo apt-get install -y nodejs - -For Centos / Red Hat use this source: - - curl -sL https://rpm.nodesource.com/setup_22.x | sudo bash - - sudo yum install -y nodejs +```bash +curl -sL https://deb.nodesource.com/setup_22.x | sudo bash - +sudo apt-get install -y nodejs +``` -Upgrade to npm 10+: +For CentOS or Red Hat, use this source: - npm install --global npm +```bash +curl -sL https://rpm.nodesource.com/setup_22.x | sudo bash - +sudo yum install -y nodejs +``` ## Java -Optional. - -Java is only for the [JDBC proxy](/developer/development/#connectors) connector which is not built automatically anymore. - -**Install Oracle JDK** - -On Ubuntu 16.04 or less only: - - sudo add-apt-repository ppa:webupd8team/java - sudo apt-get update - sudo apt-get install oracle-java8-installer - -On Centos: - -* [Oracle JDK](https://www.digitalocean.com/community/tutorials/how-to-install-java-on-centos-and-fedora) - -On Mac: - - brew cask install adoptopenjdk +Java is optional. It is only needed for the [JDBC proxy](/developer/development/#connectors) connector, which is not built by default. Install a JDK and Maven from your OS package manager if you need it. -## Supported Browsers +## Supported browsers -The two latest LTS versions of each browsers: +The two latest LTS versions of each browser: * Edge * Safari diff --git a/docs/docs-site/content/administrator/installation/install.md b/docs/docs-site/content/administrator/installation/install.md index 804a518d583..29c7825dd97 100644 --- a/docs/docs-site/content/administrator/installation/install.md +++ b/docs/docs-site/content/administrator/installation/install.md @@ -5,16 +5,52 @@ draft: false weight: 2 --- -Start from a [release](/releases/) tarball or [building](https://github.com/cloudera/hue#getting-started) the project yourself. +Build Hue from a clone of the [repository](https://github.com/cloudera/hue) and install it on your server. -Make sure you have the [dependencies](/administrator/installation/dependencies/) for your operating system. +## Prerequisites -Then configure `$PREFIX` with the path where you want to install Hue by running: +* The [dependencies](/administrator/installation/dependencies/) for your operating system: Python 3.11 and Node.js 22.13 or later. +* A user to run Hue. Run Hue as a non-root user. - PREFIX=/usr/share make install - cd /usr/share/hue +## Install Hue -You can install Hue anywhere on your system, and run Hue as a non-root user. +1. Set the build variables: -It is a good practice to create a new user for Hue and either install Hue in -that user's home directory, or in a directory within `/usr/share`. + ```bash + export PYTHON_VER=python3.11 + ``` + +2. Set `PREFIX` to the directory that will contain the `hue` directory, then install: + + ```bash + PREFIX=$HOME make install + cd $HOME/hue + ``` + + Hue is installed in `$PREFIX/hue`. If you do not set `PREFIX`, it defaults to `/usr/local`, which usually requires root. To install in a shared location such as `/usr/share`, run the command with `sudo` and give the Hue user ownership of the `hue` directory. + +You can install Hue anywhere on your system. Install it in a directory within `/usr/share` or in the home directory of the user that runs Hue. + +## Build a tarball + +To build a tarball for production, run `make prod` in the Hue directory: + +```bash +make prod +``` + +The tarball is created in `build/release/prod/`. + +## Start Hue + +After the installation, start Hue from the installation directory: + +```bash +build/env/bin/supervisor +``` + +The supervisor starts the Hue processes. Open `http://:8888/`. The first user who logs in becomes the superuser. + +> **Note:** Hue uses an embedded SQLite database by default. [Configure another database](/administrator/administration/database/) before real use. + +Next, [configure the server](/administrator/configuration/) and point it to the external services to query. diff --git a/docs/docs-site/content/developer/_index.md b/docs/docs-site/content/developer/_index.md index be45efb26fc..38e11551d3b 100644 --- a/docs/docs-site/content/developer/_index.md +++ b/docs/docs-site/content/developer/_index.md @@ -6,10 +6,10 @@ chapter = false pre = "3. " +++ -Those are the main categories of development: +These are the main categories of development: -* **[Develop](/developer/development)**: Ramping-up with on the Hue development service and SQL parsers -* **[Components](/developer/components)**: Packaged library of reusable SQL parsers and SQL Editor -* **[API](/developer/api)**: Directly interact with the API server +* **[Develop](/developer/development)**: Ramp up on the Hue development environment and the SQL parsers +* **[Components](/developer/components)**: Reusable packages, such as the SQL parsers and the SQL Editor +* **[API](/developer/api)**: Call the server with the REST API or the Python shell -Also check on how to get started on [contribution](https://github.com/cloudera/hue/blob/master/CONTRIBUTING.md) ideas. +To contribute, read the [contribution guide](https://github.com/cloudera/hue/blob/master/CONTRIBUTING.md). diff --git a/docs/docs-site/content/developer/api/_index.md b/docs/docs-site/content/developer/api/_index.md index 41350b1df1f..51094bbfa89 100644 --- a/docs/docs-site/content/developer/api/_index.md +++ b/docs/docs-site/content/developer/api/_index.md @@ -4,6 +4,14 @@ draft: false weight: 4 --- -Interact with the API via [REST](/developer/api/rest/), [Python](/developer/api/python/) or the [CLI](/developer/api/cli/). +Call Hue from your own code with one of these interfaces. + +## REST + +Run queries, manage files, import data and administer connectors over HTTP. Read the [REST API](/developer/api/rest/). + +## Python + +Run code in the Hue shell on the server, for example to manage users and documents. Read the [Python API](/developer/api/python/). The JavaScript API from the [NPM package](/developer/components/) is not documented yet. diff --git a/docs/docs-site/content/developer/api/cli/_index.md b/docs/docs-site/content/developer/api/cli/_index.md deleted file mode 100644 index ef7feb46c54..00000000000 --- a/docs/docs-site/content/developer/api/cli/_index.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: "CLI" -date: 2020-09-08T00:00:00-07:00 -draft: false -weight: 2 ---- - -Easily leverage the Query Service via the CLI. - -- Execute an SQL statement or saved query -- List, download, upload files - - -Install from https://pypi.org/project/gethue/: - - pip install gethue - -And use: - - > compose --help - - > compose auth - - Api url [https://demo.gethue.com]: - Username [demo]: - Password [demo]: - Auth: success 200 - Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjMxMjE5MDkxLCJqdGkiOiJkNGJkY2Q5M2NjMjg0MDlkYWJlYWZhNGRlNjlkOTMzMyIsInVzZXJfaWQiOjJ9.Gr8bW_JaZ8yzQ3eEZYp3jKbdsSgLAXxqvSRbeU6jhLg - -List remote storage files: - - compose storage list s3a://demo-gethue diff --git a/docs/docs-site/content/developer/api/python/_index.md b/docs/docs-site/content/developer/api/python/_index.md index b84ea1708e0..ea688007d9a 100644 --- a/docs/docs-site/content/developer/api/python/_index.md +++ b/docs/docs-site/content/developer/api/python/_index.md @@ -5,197 +5,190 @@ draft: false weight: 3 --- -Leverage the built-in Python shell to interact with the server and the API. +Use the Hue Python shell to run code with the server settings and database. Use it for administration tasks such as managing users and documents. - build/env/bin/hue shell +Start the shell from your Hue installation: -If using Cloudera Manager, as a *root* user launch the shell. - -Export the configuration directory: - - export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print $9}'`" - echo $HUE_CONF_DIR - > /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER - -Get the process id: - - lsof -i :8888|grep -m1 hue|awk '{ print $2 }' - > 14850 - -In order to export all Hue's env variables: - - for line in `strings /proc/$(lsof -i :8888|grep -m1 hue|awk '{ print $2 }')/environ|egrep -v "^HOME=|^TERM=|^PWD="`;do export $line;done - -And finally launch the shell by: - - HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 /opt/cloudera/parcels/CDH/lib/hue/build/env/bin/hue shell - > ALERT: This appears to be a CM Managed environment - > ALERT: HUE_CONF_DIR must be set when running hue commands in CM Managed environment - > ALERT: Please run 'hue --cm-managed' +```bash +build/env/bin/hue shell +``` +In the examples below, `user` is a Django user object, for example `User.objects.get(username='demo')`. ## Query -Requires a user object: +Run a statement and wait for it to finish. The call needs a request object with a user: + +```python +from django.contrib.auth.models import User +from notebook.models import make_notebook, MockRequest - from notebook.models import make_notebook, MockRequest +user = User.objects.get(username='demo') - job = make_notebook( - name='List tables in Salesforce database', - editor_type='hive', - statement='SHOW TABLES', - status='ready', - database='sfdc', - on_success_url='assist.db.refresh' - ) +job = make_notebook( + name='List tables in the sales database', + editor_type='hive', + statement='SHOW TABLES', + status='ready', + database='sales', + on_success_url='assist.db.refresh' +) - request = MockRequest(user=user) - job.execute_and_wait(request) +request = MockRequest(user=user) +job.execute_and_wait(request) +``` ## Storage -### S3 +Get a client for a storage with `get_client(name, fs, user)`. The `fs` argument is the scheme of the storage, for example `s3a` or `abfs`. -Interact directly with S3 by first getting the client: +### S3 - from desktop.lib.fsmanager import get_client +Get the client: - s3fs = get_client('default', 's3a', 'csso_hueuser') +```python +from desktop.lib.fsmanager import get_client -Then grab the key: +s3fs = get_client('default', 's3a', 'demo') +``` - k = s3fs._get_key('s3a://gethue/') - k.exists() - s3fs.stats('s3a://gethue/user/gethue/footravel.csv').to_json_dict() +Then get a key: -Or perform various FS operations: +```python +k = s3fs._get_key('s3a://gethue/') +k.exists() +s3fs.stats('s3a://gethue/user/gethue/footravel.csv').to_json_dict() +``` - b = s3fs._get_bucket('gethue') - list(b.list(prefix='user/gethue/')) +Or run file system operations: - new_k = b.new_key('user/gethue/data/3') - new_k.set_contents_from_string('123') +```python +b = s3fs._get_bucket('gethue') +list(b.list(prefix='user/gethue/')) - # new_k.delete() - result = new_k.bucket.delete_keys(new_k) +new_k = b.new_key('user/gethue/data/3') +new_k.set_contents_from_string('123') - k = s3fs._get_key('s3a://gethue/user/gethue') +# Delete the key +result = new_k.bucket.delete_keys(new_k) - s3fs.listdir_stats('s3a://gethue/user/gethue') - s3fs.mkdir('s3a://gethue/user/gethue/demo') +s3fs.listdir_stats('s3a://gethue/user/gethue') +s3fs.mkdir('s3a://gethue/user/gethue/demo') +``` ### ADLS +Get the client: -Interact directly with ADLS by first getting the client: - - from desktop.lib.fsmanager import get_client +```python +from desktop.lib.fsmanager import get_client - fs = get_client('default', 'abfs', 'csso_hueuser') +fs = get_client('default', 'abfs', 'demo') +``` -Perform various FS operations: +Then run file system operations: - # Stats - fs.stats('abfs://data/user/csso_hueuser/demo_dir') +```python +# Stats +fs.stats('abfs://data/user/demo/demo_dir') - # List directory - fs.listdir('abfs://data/user/csso_hueuser/demo_dir') +# List a directory +fs.listdir('abfs://data/user/demo/demo_dir') - # Create directory - fs.mkdir('abfs://data/user/csso_hueuser/new_dir') +# Create a directory +fs.mkdir('abfs://data/user/demo/new_dir') - # Create file - fs.create('abfs://data/user/csso_hueuser/demo_dir/newfile.txt') +# Create a file +fs.create('abfs://data/user/demo/demo_dir/newfile.txt') - # Create file with write data - fs.create('abfs://data/user/csso_hueuser/demo_dir/demo_file.txt', data='Hello world!') +# Create a file with content +fs.create('abfs://data/user/demo/demo_dir/demo_file.txt', data='Hello world!') - # Read - fs.read('abfs://data/user/csso_hueuser/demo_dir/demo_file.txt') +# Read a file +fs.read('abfs://data/user/demo/demo_dir/demo_file.txt') - # Delete path (can be file or empty directory) - fs.remove('abfs://data/user/csso_hueuser/demo_dir/demo_file.txt') +# Delete a file or an empty directory +fs.remove('abfs://data/user/demo/demo_dir/demo_file.txt') - # Delete directory with recursive as true - fs.rmtree('abfs://data/user/csso_hueuser/demo_dir') +# Delete a directory and its content +fs.rmtree('abfs://data/user/demo/demo_dir') - # Chmod (accepts both octal number and string) - fs.chmod('abfs://data/user/csso_hueuser/demo_dir/demo_file.txt', permissionNumber='777') +# Change the permissions (octal number or string) +fs.chmod('abfs://data/user/demo/demo_dir/demo_file.txt', permissionNumber='777') - # Rename path - fs.rename('abfs://data/user/csso_hueuser/old_name_dir', 'abfs://data/user/csso_hueuser/new_name_dir') +# Rename a path +fs.rename('abfs://data/user/demo/old_name_dir', 'abfs://data/user/demo/new_name_dir') - # Copy (for both file and directory) - fs.copy('abfs://data/user/csso_hueuser/source_path', 'abfs://data/user/csso_hueuser/destination_path') +# Copy a file or a directory +fs.copy('abfs://data/user/demo/source_path', 'abfs://data/user/demo/destination_path') +``` ## Users -### Create +### Create a user - from desktop.auth.backend import create_user +```python +from desktop.auth.backend import create_user - bob = create_user(username='bob', password='secret1', is_superuser=True) - alice = create_user(username='alice', password='secret2', is_superuser=False) +bob = create_user(username='bob', password='secret1', is_superuser=True) +alice = create_user(username='alice', password='secret2', is_superuser=False) +``` -### Find or Create +### Find or create a user - from desktop.auth.backend import find_or_create_user +```python +from desktop.auth.backend import find_or_create_user - bob = find_or_create_user(username='bob', password='secret1', is_superuser=True) - alice = find_or_create_user(username='alice', password='secret2', is_superuser=False) +bob = find_or_create_user(username='bob', password='secret1', is_superuser=True) +alice = find_or_create_user(username='alice', password='secret2', is_superuser=False) +``` -### Convert to admin +### Make a user an admin -Then type something similar to: +```python +from django.contrib.auth.models import User - from django.contrib.auth.models import User +a = User.objects.get(username='hdfs') +a.is_staff = True +a.is_superuser = True +a.set_password('my_secret') +a.save() +``` - a = User.objects.get(username='hdfs') - a.is_staff = True - a.is_superuser = True - a.set_password('my_secret') - a.save() +### Change a password -### Changing password +```python +from django.contrib.auth.models import User - from django.contrib.auth.models import User +user = User.objects.get(username='example') +user.set_password('some password') +user.save() +``` - user = User.objects.get(username='example') - user.set_password('some password') - user.save() +## Documents -### Counting documents +### Count the documents of a user - from django.contrib.auth.models import User - from desktop.models import Document2 +The `perms` argument is `both` (default), `shared` or `owned`. - user = User.objects.get(username='demo') - Document2.objects.documents(user=user).count() +```python +from django.contrib.auth.models import User +from desktop.models import Document2 - In [8]: Document2.objects.documents(user=user).count() - Out[8]: 1167 +user = User.objects.get(username='demo') - In [10]: Document2.objects.documents(user=user, perms='own').count() - Out[10]: 1166 +# Owned and shared documents +Document2.objects.documents(user=user).count() - In [11]: Document2.objects.documents(user=user, perms='own', include_history=True).count() - Out[11]: 7125 +# Owned documents only +Document2.objects.documents(user=user, perms='owned').count() - In [12]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True).count() - Out[12]: 7638 +# Include the query history +Document2.objects.documents(user=user, perms='owned', include_history=True).count() - In [13]: Document2.objects.documents(user=user, perms='own', include_history=True, include_trashed=True, include_managed=True).count() - Out[13]: 31408 +# Include the trashed documents +Document2.objects.documents(user=user, perms='owned', include_history=True, include_trashed=True).count() - Out[14]: - (85667L, - {u'desktop.Document': 18524L, - u'desktop.Document2': 31409L, - u'desktop.Document2Permission': 556L, - u'desktop.Document2Permission_groups': 277L, - u'desktop.Document2Permission_users': 0L, - u'desktop.Document2_dependencies': 15087L, - u'desktop.DocumentPermission': 1290L, - u'desktop.DocumentPermission_groups': 0L, - u'desktop.DocumentPermission_users': 0L, - u'desktop.Document_tags': 18524L}) +# Include the documents managed by Hue +Document2.objects.documents(user=user, perms='owned', include_history=True, include_trashed=True, include_managed=True).count() +``` diff --git a/docs/docs-site/content/developer/api/rest/_index.md b/docs/docs-site/content/developer/api/rest/_index.md index d03270eb3c6..3b0209afe9c 100644 --- a/docs/docs-site/content/developer/api/rest/_index.md +++ b/docs/docs-site/content/developer/api/rest/_index.md @@ -5,951 +5,1007 @@ draft: false weight: 1 --- -Interact with the Query server (e.g. submit a SQL query, upload some files in cloud storages, search for a table...) via a REST API. +Use the REST API to run SQL queries, manage files in the configured storages, import files, search for tables and manage connectors. -Users authenticate with the same credentials as they would do in the Browser login page. +The examples use these placeholders: -## Quickstart +```bash +export HUE_URL=https://hue.example.com +export TOKEN= +``` -The API can be called directly via REST. +## Quickstart -First [authenticate](/developer/api/rest/#authentication) with your account credentials and get a token. Then provide the token in all following requests as header, e.g. +1. [Authenticate](#authentication) and get an access token. +2. Send the token in the `Authorization` header of every request. - curl -X POST https://demo.gethue.com/api/v1/editor/execute/hive --data 'statement=SHOW TABLES' -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM5NjMxLCJqdGkiOiI0NTY3NTA4MzM5YjY0MjFmYTMzZDJjMzViZWUyMDAyMCIsInVzZXJfaWQiOjF9.qrMNrr69eo38dOsV2aYp8k6WqBeyJZkbSuavxA_o_kM" +```bash +curl -X POST "$HUE_URL/api/v1/token/auth/" \ + -H "Content-Type: application/json" \ + -d '{"username": "demo", "password": "demo"}' -The default content type is form data, e.g.: +curl -X POST "$HUE_URL/api/v1/editor/execute/hive" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'statement=SHOW TABLES' +``` - -H "Content-Type: application/x-www-form-urlencoded" -d 'username=demo&password=demo' +Read the full [query example](#execute-a-query) below. -It is possible to submit data in JSON format for the calls also reading the data via `request.body`: +### Python - -H "Content-Type: application/json" -d '{"username": "demo", "password": "demo"}' +Install the [requests](https://pypi.org/project/requests/) library: -### Curl +```bash +pip install requests +``` -Calling without credentials: +Then authenticate and run a statement: - curl -X POST https://demo.gethue.com/api/v1/query/create_notebook -H "Content-Type: application/json" +```python +import time - {"detail":"Authentication credentials were not provided."} +import requests -Authenticating and getting a [JWT token](https://jwt.io/): +HUE_URL = "https://hue.example.com" - curl -X POST https://demo.gethue.com/api/v1/token/auth/ -H "Content-Type: application/json" -d '{"username": "demo", "password": "demo"}' +response = requests.post( + f"{HUE_URL}/api/v1/token/auth/", + json={"username": "demo", "password": "demo"}, +) +response.raise_for_status() +headers = {"Authorization": f"Bearer {response.json()['access']}"} - {"refresh":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTYyMTcyNDYzMSwianRpIjoiOGM0NDRjYzRhN2VhNGMxZDliMGZhNmU1YzUyMjM1MjkiLCJ1c2VyX2lkIjoxfQ.t6t7_eYrNhpGN3-Jz5MDLXM8JtGP7V9Y9lacOTInqqQ","access":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM4NTMxLCJqdGkiOiJhZjgwN2E0ZjBmZDI0ZWMxYWQ2NTUzZjEyMjIyYzU4YyIsInVzZXJfaWQiOjF9.dQ1P3hbzSytp9-o8bWlcOcwrdwRVy95M2Eolph92QMA"} +response = requests.post( + f"{HUE_URL}/api/v1/editor/execute/hive", + headers=headers, + data={"statement": "SELECT 1, 2, 3"}, +) +operation_id = response.json()["history_uuid"] -Re-using the token when making actual calls: +while True: + response = requests.post( + f"{HUE_URL}/api/v1/editor/check_status", + headers=headers, + data={"operationId": operation_id}, + ) + if response.json()["query_status"]["status"] not in ("waiting", "running"): + break + time.sleep(1) + +response = requests.post( + f"{HUE_URL}/api/v1/editor/fetch_result_data", + headers=headers, + data={"operationId": operation_id}, +) +print(response.json()["result"]["data"]) +``` - curl -X POST https://demo.gethue.com/api/v1/query/create_notebook -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM5NjMxLCJqdGkiOiI0NTY3NTA4MzM5YjY0MjFmYTMzZDJjMzViZWUyMDAyMCIsInVzZXJfaWQiOjF9.qrMNrr69eo38dOsV2aYp8k6WqBeyJZkbSuavxA_o_kM" +### JavaScript - {"status": 0, "notebook": {"name": "My Notebook", "uuid": "1e23314f-b01e-4c18-872f-dc143475f063", "description": "", "type": "notebook", "isSaved": false, "isManaged": false, "skipHistorify": false, "sessions": [], "snippets": [], "directoryUuid": null}} +This example uses [Axios](https://axios-http.com/): -### Python +```javascript +const axios = require('axios'); -In this code snippet, we will use the [requests](https://pypi.org/project/requests/) library: +const client = axios.create({ baseURL: 'https://hue.example.com' }); - pip install requests +async function main() { + const auth = await client.post('/api/v1/token/auth/', { username: 'demo', password: 'demo' }); + client.defaults.headers.common['Authorization'] = `Bearer ${auth.data.access}`; -And then: + const types = await client.get('/api/v1/connector/types'); + console.log(types.data); +} - import json - import requests +main(); +``` - session = requests.Session() +## Conventions - data = { - 'username': 'demo', - 'password': 'demo', - } +- All paths start with `/api/v1/`. A trailing slash is optional, except for `connector/instance/new//` and `connector/instance/get/`, which must not end with one. +- Requests are form data (`application/x-www-form-urlencoded`) unless the endpoint says otherwise. The token endpoints also accept JSON. `storage/rename` requires a JSON body. +- The editor, connector and catalog endpoints return `200 OK` with a body containing `"status": 0` on success. On failure, `status` is not 0 (`-1` is the generic error, other values include `-2` for an expired session and `1` for a query error) and `message` holds the reason. +- The storage endpoints return plain HTTP status codes. On failure, the body is an error message or `{"error": ""}`. +- A request without valid credentials returns `401 Unauthorized`: - response = session.post("https://demo.gethue.com/api/v1/token/auth", data=data) - print('Auth: %s %s' % ('success' if response.status_code == 200 else 'error', response.status_code)) +```json +{"detail": "Authentication credentials were not provided."} +``` - token = json.loads(response.content)['access'] - print('Token: %s' % token) +- The server publishes its OpenAPI schema at `/api/schema` and an interactive Swagger page at `/api/swagger`. - response = requests.post( - 'https://demo.gethue.com/api/v1/query/autocomplete', - headers={ - 'Authorization': 'Bearer %s' % token, - "Content-Type": "application/x-www-form-urlencoded" - }, - data={'snippet': json.dumps({"type":"1"})} - ) - print(response.status_code) - print(response.text) +## Authentication -### JavaScript +The API accepts these authentication methods, in order of priority. The administrator can change the list with `api_auth` in the `[desktop] [[auth]]` section of `hue.ini`. -In the meantime, with Axios: +1. **JWT access token** in the `Authorization: Bearer ` header. +2. **Session cookie** from a browser login. - +The token endpoints check your credentials against the same [authentication backends](/administrator/configuration/server/#authentication) as the login page. - +```bash +curl -X POST "$HUE_URL/api/v1/token/auth/" \ + -H "Content-Type: application/json" \ + -d '{"username": "demo", "password": "demo"}' +``` -## Authentication +```json +{"refresh": "", "access": ""} +``` -The API authenticates via the [authentication backends](/administrator/configuration/server/#authentication) of the server (same as going via the login page). This is consistent and users are free to interact via their browsers or API. +An access token is valid for one day. A refresh token is valid for seven days. -Then a JWT token is returned and needs to be passed as a bearer in the headers for all the API calls. +**Errors:** `401 Unauthorized` when the credentials are wrong. -**Wrong credentials**: on bad authentication, it will return a `401 unauthorized` response, e.g.: +### Use a token - curl -X POST https://demo.gethue.com/api/v1/editor/create_notebook -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM5NjMxLCJqdGkiOiI0NTY3NTA4MzM5YjY0MjFmYTMzZDJjMzViZWUyMDAyMCIsInVzZXJfaWQiOjF9.qrMNrr69eo38dOsV2aYp8k6WqBeyJZkbSuavxA_o_kM" +Send the `access` value as a bearer token: - {"detail":"Given token not valid for any token type","code":"token_not_valid","messages":[{"token_class":"AccessToken","token_type":"access","message":"Token is invalid or expired"}]} +```bash +curl -X POST "$HUE_URL/api/v1/query/create_notebook" \ + -H "Authorization: Bearer $TOKEN" +``` - [09/Jul/2021 23:58:40 -0700] access INFO demo.gethue.com -anon- - "POST /api/v1/editor/create_notebook HTTP/1.1" returned in 2ms 401 183 (mem: 124mb) +```json +{"status": 0, "notebook": {"name": "My Notebook", "uuid": "1e23314f-b01e-4c18-872f-dc143475f063", "type": "notebook", "sessions": [], "snippets": []}} +``` -### Authenticate +An invalid or expired token returns `401 Unauthorized`: -Provide login credentials and get a [JWT token](https://jwt.io/): +```json +{"detail": "Given token not valid for any token type", "code": "token_not_valid", "messages": [{"token_class": "AccessToken", "token_type": "access", "message": "Token is invalid or expired"}]} +``` - curl -X POST https://demo.gethue.com/api/v1/token/auth -d 'username=demo&password=demo' +### Verify a token - {"refresh":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTYyMTcyNDYzMSwianRpIjoiOGM0NDRjYzRhN2VhNGMxZDliMGZhNmU1YzUyMjM1MjkiLCJ1c2VyX2lkIjoxfQ.t6t7_eYrNhpGN3-Jz5MDLXM8JtGP7V9Y9lacOTInqqQ","access":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM4NTMxLCJqdGkiOiJhZjgwN2E0ZjBmZDI0ZWMxYWQ2NTUzZjEyMjIyYzU4YyIsInVzZXJfaWQiOjF9.dQ1P3hbzSytp9-o8bWlcOcwrdwRVy95M2Eolph92QMA"} +**Method and URL:** `POST /api/v1/token/verify/` -And keep the `access` token as the value of the bearer header in the API calls. +| Name | Type | Required | Description | +|------|------|----------|-------------| +| token | string | Yes | The token to check. | -### Validate token +```bash +curl -X POST "$HUE_URL/api/v1/token/verify/" -d "token=$TOKEN" +``` -The validity (i.e. did it expire?) of an `access` token can be verified: +A valid token returns `200 OK` and `{}`. An invalid or expired token returns `401 Unauthorized`. - curl -X POST https://demo.gethue.com/api/v1/token/verify/ -d 'token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjIxNjM4NTMxLCJqdGkiOiJhZjgwN2E0ZjBmZDI0ZWMxYWQ2NTUzZjEyMjIyYzU4YyIsInVzZXJfaWQiOjF9.dQ1P3hbzSytp9-o8bWlcOcwrdwRVy95M2Eolph92QMA' +### Refresh a token -### Refresh token +**Method and URL:** `POST /api/v1/token/refresh/` -Similarly, an `access` token validity can be extended via a refresh sending the `refresh` token obtained in the initial authentication. +| Name | Type | Required | Description | +|------|------|----------|-------------| +| refresh | string | Yes | The `refresh` token from the authentication response. | - curl -X POST https://demo.gethue.com/api/v1/token/refresh/ -d 'refresh=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTYyMTcyNDYzMSwianRpIjoiOGM0NDRjYzRhN2VhNGMxZDliMGZhNmU1YzUyMjM1MjkiLCJ1c2VyX2lkIjoxfQ.t6t7_eYrNhpGN3-Jz5MDLXM8JtGP7V9Y9lacOTInqqQ' +```bash +curl -X POST "$HUE_URL/api/v1/token/refresh/" -d "refresh=" +``` -### Custom JWT Authentication +```json +{"access": ""} +``` -Users can authenticate with their own JWT with the help of custom backend **(supporting RSA256)**. To enable it, add the following in the `hue.ini`: +**Errors:** `401 Unauthorized` when the refresh token is invalid or expired. - [desktop] - [[auth]] - [[[jwt]]] - key_server_url=https://ext_authz:8000 - issuer= - audience=hue - username_header=sub +### Session cookie -Also, to allow Hue to send this JWT to external services like Impala, enable the following flag in `hue.ini`: +If you call the API from a browser session, Hue also accepts the session cookie. For `POST` requests, send the value of the `csrftoken` cookie in the `X-CSRFToken` header. Without it, the request fails with `403 Forbidden`. Bearer token requests do not need the CSRF header. - [desktop] - use_thrift_http_jwt=true +### Custom JWT -If you wish to implement your own custom auth (having customized connection with external auth server or using different signing algorithm etc.), then you can follow the [Django REST Framework custom pluggability](https://www.django-rest-framework.org/api-guide/authentication/#custom-authentication) and add like this [dummy auth](https://github.com/cloudera/hue/blob/d75c8fc7b307fc67ef9a2a58e36cfb4ace6cd461/desktop/core/src/desktop/auth/api_authentications.py#L119). +You can authenticate with a JWT issued by your own identity provider. Hue verifies RS256 signatures. Add this to `hue.ini`: -And then, add it in `hue.ini` (comma separated and in order of priority if multiple auth backends present): +```ini +[desktop] +[[auth]] +[[[jwt]]] +key_server_url=https://ext_authz:8000 +issuer= +audience=hue +username_header=sub +``` - [desktop] - [[auth]] - api_auth= +To forward the JWT to services such as Impala, also set: -## SQL Querying +```ini +[desktop] +use_thrift_http_jwt=true +``` -### Execute a Query +You can also write your own [Django REST Framework authentication class](https://www.django-rest-framework.org/api-guide/authentication/#custom-authentication). Then list it in `hue.ini`, separated by commas if you use several: -Now that we are authenticated, here is how to execute a `SHOW TABLES` SQL query via the `hive` connector. You could repeat the steps with any query you want, e.g. `SELECT * FROM web_logs LIMIT 100`. +```ini +[desktop] +[[auth]] +api_auth= +``` -Selecting the **dialect** argument in `/api/v1/editor/execute/`: -- **hive**: select the configured Hive dialect -- **1**: select the connector id 1 -- **hive-1**: select the interpreter id 1 and hints that it is a Hive dialect -- If blank will pick the first interpreter +## SQL querying -Optional parameter: -- **database**: select a specific database +### Execute a query -For a `SHOW TABLES`, first we send the query statement: +Submit a statement to a dialect. - curl -X POST https://demo.gethue.com/api/v1/editor/execute/hive --data 'statement=SHOW TABLES' +**Method and URL:** `POST /api/v1/editor/execute/` - {"status": 0, "history_id": 17880, "handle": {"statement_id": 0, "session_type": "hive", "has_more_statements": false, "guid": "EUI32vrfTkSOBXET6Eaa+A==\n", "previous_statement_hash": "3070952e55d733fb5bef249277fb8674989e40b6f86c5cc8b39cc415", "log_context": null, "statements_count": 1, "end": {"column": 10, "row": 0}, "session_id": 63, "start": {"column": 0, "row": 0}, "secret": "RuiF0LEkRn+Yok/gjXWSqg==\n", "has_result_set": true, "session_guid": "c845bb7688dca140:859a5024fb284ba2", "statement": "SHOW TABLES", "operation_type": 0, "modified_row_count": null}, "history_uuid": "63ce87ba-ca0f-4653-8aeb-e9f5c1781b78"} +| Name | Type | Required | Description | +|------|------|----------|-------------| +| dialect | path | No | The configured dialect, for example `hive`. Use `-`, for example `hive-1`, to pick an interpreter by id. If you omit it, Hue uses the first interpreter. | +| statement | string | Yes | The SQL statement. Do not use double quotes or line breaks in it. | +| database | string | No | The database to run the statement in. | -Then check the operation (its value is **history_uuid** from the execute response) status until its result is ready to fetch: +```bash +curl -X POST "$HUE_URL/api/v1/editor/execute/hive" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'statement=SHOW TABLES' +``` - curl -X POST https://demo.gethue.com/api/v1/editor/check_status --data 'operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78' +```json +{"status": 0, "history_id": 17880, "history_uuid": "63ce87ba-ca0f-4653-8aeb-e9f5c1781b78", "handle": {"statement_id": 0, "session_type": "hive", "has_more_statements": false, "statements_count": 1, "has_result_set": true, "statement": "SHOW TABLES", "operation_type": 0, "modified_row_count": null}} +``` - {"status": 0, "query_status": {"status": "available", "has_result_set": true}} +Keep the `history_uuid` value. It is the `operationId` for the next calls. -And now ask for the resultset of the statement: +**Errors:** `200 OK` with a non-zero `status` and a `message`, for example `{"status": -1, "message": ""}`. - curl -X POST https://demo.gethue.com/api/v1/editor/fetch_result_data --data 'operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78' +### Check the status - {"status": 0, "result": {"has_more": true, "type": "table", "meta": [{"comment": "from deserializer", "type": "STRING_TYPE", "name": "tab_name"}], "data": [["adavi"], ["adavi1"], ["adavi2"], ["ambs_feed"], ["apx_adv_deduction_data_process_total"], ["avro_table"], ["avro_table1"], ["bb"], ["bharath_info1"], ["bucknew"], ["bucknew1"], ["chungu"], ["cricket3"], ["cricket4"], ["cricket5_view"], ["cricketer"], ["cricketer_view"], ["cricketer_view1"], ["demo1"], ["demo12345"], ["dummy"], ["embedded"], ["emp"], ["emp1_sept9"], ["emp_details"], ["emp_sept"], ["emp_tbl1"], ["emp_tbl2"], ["empdtls"], ["empdtls_ext"], ["empdtls_ext_v2"], ["employee"], ["employee1"], ["employee_ins"], ["empppp"], ["events"], ["final"], ["flight_data"], ["gopalbhar"], ["guruhive_internaltable"], ["hell"], ["info1"], ["lost_messages"], ["mnewmyak"], ["mortality"], ["mscda"], ["myak"], ["mysample"], ["mysample1"], ["mysample2"], ["network"], ["ods_t_exch_recv_rel_wfz_stat_szy"], ["olympicdata"], ["p_table"], ["partition_cricket"], ["partitioned_user"], ["s"], ["sample"], ["sample_07"], ["sample_08"], ["score"], ["stg_t_exch_recv_rel_wfz_stat_szy"], ["stocks"], ["students"], ["studentscores"], ["studentscores2"], ["t1"], ["table_name"], ["tablex"], ["tabley"], ["temp"], ["test1"], ["test2"], ["test21"], ["test_info"], ["topage"], ["txnrecords"], ["u_data"], ["udata"], ["user_session"], ["user_test"], ["v_empdtls"], ["v_empdtls_ext"], ["v_empdtls_ext_v2"], ["web_logs"]], "isEscaped": true}} +Poll until the result is ready. -And if we wanted to get the execution log for this statement: +**Method and URL:** `POST /api/v1/editor/check_status` - curl -X POST https://demo.gethue.com/api/v1/editor/get_logs --data 'operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78' +| Name | Type | Required | Description | +|------|------|----------|-------------| +| operationId | string | Yes | The `history_uuid` returned by the execute call. | - {"status": 0, "progress": 5, "jobs": [], "logs": "", "isFullLogs": false} +```bash +curl -X POST "$HUE_URL/api/v1/editor/check_status" \ + -H "Authorization: Bearer $TOKEN" \ + -d "operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78" +``` -Same but in Python: +```json +{"status": 0, "query_status": {"status": "available", "has_result_set": true}} +``` - params = { - 'statement': 'SELECT 1, 2, 3', - } +The query is ready when `query_status.status` is `available`. While it runs, the value is `waiting` or `running`. - response = requests.post( - 'https://demo.gethue.com/api/v1/editor/execute/mysql', - headers={ - 'Authorization': 'Bearer %s' % token, - "Content-Type": "application/x-www-form-urlencoded" - }, - data=params - ) - print(response.status_code) - print(response.text) +### Fetch the results - resp_content = json.loads(response.text) +**Method and URL:** `POST /api/v1/editor/fetch_result_data` - data = { - 'operationId': resp_content['history_uuid'], - } +| Name | Type | Required | Description | +|------|------|----------|-------------| +| operationId | string | Yes | The `history_uuid` returned by the execute call. | +| rows | integer | No | Number of rows to fetch. The default is 100. | +| startOver | boolean | No | Set to `true` to read from the first row again. The default is `false`. | - response = requests.post( - 'https://demo.gethue.com/api/v1/editor/check_status', - headers={ - 'Authorization': 'Bearer %s' % token, - "Content-Type": "application/x-www-form-urlencoded" - }, - data=data - ) - print(response.status_code) - print(response.text) +```bash +curl -X POST "$HUE_URL/api/v1/editor/fetch_result_data" \ + -H "Authorization: Bearer $TOKEN" \ + -d "operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78" -d "rows=100" +``` +```json +{"status": 0, "result": {"has_more": true, "type": "table", "meta": [{"comment": "from deserializer", "type": "STRING_TYPE", "name": "tab_name"}], "data": [["web_logs"], ["sample_07"]], "isEscaped": true}} +``` - response = requests.post( - 'https://demo.gethue.com/api/v1/editor/fetch_result_data', - headers={ - 'Authorization': 'Bearer %s' % token, - "Content-Type": "application/x-www-form-urlencoded" - }, - data=data - ) - print(response.status_code) - print(response.text) +Call it again to get the next rows while `has_more` is `true`. -### Listing Databases +The related calls `POST /api/v1/editor/fetch_result_metadata` and `POST /api/v1/editor/fetch_result_size` take the same `operationId` parameter. - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete/ -d 'snippet={"type":"hive"}' +### Get the logs - {"status": 0, "databases": ["default", "information_schema", "sys"]} +**Method and URL:** `POST /api/v1/editor/get_logs` -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +| Name | Type | Required | Description | +|------|------|----------|-------------| +| operationId | string | Yes | The `history_uuid` returned by the execute call. | +| from | integer | No | Line to start reading from. | +| size | integer | No | Number of lines to read. | -### Database details +```bash +curl -X POST "$HUE_URL/api/v1/editor/get_logs" \ + -H "Authorization: Bearer $TOKEN" \ + -d "operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78" +``` - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete// -d 'snippet={"type":"hive"}' +```json +{"status": 0, "progress": 5, "jobs": [], "logs": "", "isFullLogs": false} +``` -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +### Cancel or close a statement -Describe database API: +Use `POST /api/v1/editor/cancel_statement` to stop a running statement. Use `POST /api/v1/editor/close_statement` to release its resources after you fetched the results. Both take the `operationId` parameter. - curl -X POST https://demo.gethue.com/api/v1/editor/describe// -d 'source_type=mysql' +```bash +curl -X POST "$HUE_URL/api/v1/editor/cancel_statement" \ + -H "Authorization: Bearer $TOKEN" \ + -d "operationId=63ce87ba-ca0f-4653-8aeb-e9f5c1781b78" +``` -- **source_type:** select from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +### Get the query history -### Table details +**Method and URL:** `GET /api/v1/editor/get_history` - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete/// -d 'snippet={"type":"hive"}' +| Name | Type | Required | Description | +|------|------|----------|-------------| +| doc_type | string | Yes | The dialect, for example `hive`, `impala` or `mysql`. | +| doc_text | string | No | Filter on the name, description or statement. | +| doc_connector | string | No | Filter on a connector id. | +| page | integer | No | Page number. The default is 1. | +| limit | integer | No | Page size. The default is 50 and the maximum is 100. | -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +```bash +curl "$HUE_URL/api/v1/editor/get_history?doc_type=hive&limit=10" \ + -H "Authorization: Bearer $TOKEN" +``` -Describe table API: +```json +{"status": 0, "count": 1, "history": [{"name": "", "id": 2008, "uuid": "5b48c678-1224-4863-b523-3baab82402a7", "type": "query-hive", "data": {"statement": "CREATE TABLE w12( Name STRING, Money BIGINT )", "lastExecuted": 1621502970360, "status": "failed", "parentSavedQueryUuid": ""}, "absoluteUrl": "/editor?editor=2008"}], "message": "History fetched"} +``` - curl -X POST https://demo.gethue.com/api/v1/editor/describe//
/ -d 'source_type=1' +### Browse databases, tables and columns -- **source_type:** select from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +The autocomplete endpoint lists the objects of a dialect. All calls are `POST` requests with the same parameters. -Analyze API: +| Name | Type | Required | Description | +|------|------|----------|-------------| +| snippet | JSON string | No | Selects the dialect, for example `{"type":"hive"}`. You can use a connector id instead, for example `{"type":"1"}`. When omitted, it defaults to the connector with id `1`. | +| operation | string | No | `schema` (default), `functions` or `function`. | - curl -X POST https://demo.gethue.com/api/v1//analyze//
/ +| Purpose | URL | +|---------|-----| +| List databases | `/api/v1/editor/autocomplete/` | +| List tables of a database | `/api/v1/editor/autocomplete//` | +| List columns of a table | `/api/v1/editor/autocomplete//
/` | +| Describe a column | `/api/v1/editor/autocomplete//
//` | -- Currently supported **dialects:** impala, beeswax (hive) +```bash +curl -X POST "$HUE_URL/api/v1/editor/autocomplete/" \ + -H "Authorization: Bearer $TOKEN" \ + -d 'snippet={"type":"hive"}' +``` -Sample table data API: +```json +{"status": 0, "databases": ["default", "information_schema", "sys"]} +``` - curl -X POST https://demo.gethue.com/api/v1/editor/sample//
/ -d 'snippet={"type":"hive"}' +To list functions, add `operation=functions`. Add a database to the path for the functions of that database. To get the details of one function, use its name in the path and `operation=function`. -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +```bash +curl -X POST "$HUE_URL/api/v1/editor/autocomplete/" \ + -H "Authorization: Bearer $TOKEN" \ + -d 'snippet={"type":"hive"}' -d 'operation=functions' +``` -### Column details +### Describe a database, table or column - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete//
// -d 'snippet={"type":"hive"}' +**Method and URL:** -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +- `POST /api/v1/editor/describe//` +- `POST /api/v1/editor/describe//
/` +- `POST /api/v1/editor/describe//
/stats//` -Analyze API: +| Name | Type | Required | Description | +|------|------|----------|-------------| +| source_type | string | No | The dialect, for example `hive`, or a connector id. | - curl -X POST https://demo.gethue.com/api/v1//analyze//
// +```bash +curl -X POST "$HUE_URL/api/v1/editor/describe/default/web_logs/" \ + -H "Authorization: Bearer $TOKEN" \ + -d 'source_type=hive' +``` -- Currently supported **dialects:** impala, beeswax (hive) +### Get sample data -Sample column data API: +**Method and URL:** - curl -X POST https://demo.gethue.com/api/v1/editor/sample//
// -d 'snippet={"type":"hive"}' +- `POST /api/v1/editor/sample//
/` +- `POST /api/v1/editor/sample//
//` -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) +| Name | Type | Required | Description | +|------|------|----------|-------------| +| snippet | JSON string | No | Selects the dialect, for example `{"type":"hive"}`. When omitted, it defaults to the connector with id `1`. | -### Listing Functions +```bash +curl -X POST "$HUE_URL/api/v1/editor/sample/default/web_logs/" \ + -H "Authorization: Bearer $TOKEN" \ + -d 'snippet={"type":"hive"}' +``` -Default functions: +### Analyze a table - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete -d 'snippet={"type":"hive"}' -d 'operation=functions' +Compute the statistics of a table. -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) -- **operation:** specify the type of operation (e.g., `functions`) +**Method and URL:** -For a specific database: +- `POST /api/v1//analyze//
/` - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete/ -d 'snippet={"type":"impala"}' -d 'operation=functions' +The statistics are always computed for the whole table, even if you add a column to the URL. -- **snippet:** select the `type` from the configured dialects (e.g. `impala`) or connector IDs (e.g. `1`) -- **operation:** specify the type of operation (e.g., `functions`) +The supported dialects are `impala` and `beeswax` (Hive). Other dialects return `204 No Content`. -For a specific function/UDF details (e.g. trunc): +```bash +curl -X POST "$HUE_URL/api/v1/impala/analyze/default/web_logs/" \ + -H "Authorization: Bearer $TOKEN" +``` - curl -X POST https://demo.gethue.com/api/v1/editor/autocomplete/ -d 'snippet={"type":"hive"}' -d 'operation=function' +### Get the configuration -- **snippet:** select the `type` from the configured dialects (e.g. `hive`) or connector IDs (e.g. `1`) -- **operation:** specify the type of operation (e.g., `function`) +Get the interpreters, browsers and settings available to the current user. -### Query history +**Method and URL:** `POST /api/v1/get_config/` -We can choose a dialect for `doc_type` e.g. impala, mysql, hive, phoenix, etc. +```bash +curl -X POST "$HUE_URL/api/v1/get_config/" \ + -H "Authorization: Bearer $TOKEN" +``` - curl -X GET https://demo.gethue.com/api/v1/editor/get_history?doc_type=hive +The response is a JSON object with the keys `app_config`, `main_button_action`, `button_actions`, `default_sql_interpreter`, `cluster_type`, `clusters`, `hue_config`, `storage_browser` and `documents`, among others. The exact content depends on the server configuration. - {"status": 0, "count": 3, "history": [{"name": "", "id": 2008, "uuid": "5b48c678-1224-4863-b523-3baab82402a7", "type": "query-hive", "data": {"statement": "CREATE TABLE w12( Name STRING, Money BIGINT )", "lastExecuted": 1621502970360, "status": "failed", "parentSavedQueryUuid": ""}, "absoluteUrl": "/editor?editor=2008"}, {"name": "", "id": 2006, "uuid": "1cd32ae0-9b61-46ae-8fd4-72c4255209c3", "type": "query-hive", "data": {"statement": "CREATE TABLE q13( Name STRING, Money BIGINT )", "lastExecuted": 1621498889058, "status": "expired", "parentSavedQueryUuid": ""}, "absoluteUrl": "/editor?editor=2006"}, {"name": "", "id": 2003, "uuid": "e5ec1fa4-1a36-4e42-a814-a685b0142223", "type": "query-hive", "data": {"statement": "CREATE TABLE q11( Name STRING, Money BIGINT );\nINSERT INTO q11 VALUES ('abc', 100);", "lastExecuted": 1621498771619, "status": "expired", "parentSavedQueryUuid": ""}, "absoluteUrl": "/editor?editor=2003"}], "message": "History fetched"} +## File browsing -### Get Configuration +The [File Browser](/user/browsing/#data) lists, reads and changes files in HDFS (`hdfs://` or no prefix), S3 (`s3a://`), ADLS (`abfs://`), Google Cloud Storage (`gs://`) and Ozone (`ofs://`). Only the storages enabled on the server and allowed for your user are available. - curl -X POST https://demo.gethue.com/api/v1/get_config/ +Paths are passed in the `path` parameter, not in the URL. - {"app_config": {"editor": {"name": "editor", "displayName": "Editor", "buttonName": "Query", "interpreters": [{"name": "MySQL", "type": "mysql", "id": "mysql", "displayName": "MySQL", "buttonName": "Query", "tooltip": "Mysql Query", "optimizer": "off", "page": "/editor/?type=mysql", "is_sql": true, "is_batchable": true, "dialect": "mysql", "dialect_properties": {}}, {"name": "notebook", "type": "notebook", "displayName": "Notebook", "buttonName": "Notebook", "tooltip": "Notebook", "page": "/notebook", "is_sql": false, "dialect": "notebook"}], "default_limit": 5000, "interpreter_names": ["mysql", "notebook"], "page": "/editor/?type=mysql", "default_sql_interpreter": "mysql"}, "catalogs": [{"name": "MySQL", "type": "mysql", "id": "mysql", "displayName": "MySQL", "buttonName": "Query", "tooltip": "Mysql Query", "page": "/editor/?type=mysql", "is_sql": true, "is_catalog": true}], "browser": {"name": "browser", "displayName": "Browsers", "buttonName": "Browse", "interpreters": [{"type": "hdfs", "displayName": "Files", "buttonName": "Browse", "tooltip": "Files", "page": "/filebrowser/view=%2Fuser%2Fdemo"}, {"type": "tables", "displayName": "Tables", "buttonName": "Browse", "tooltip": "Tables", "page": "/metastore/tables"}, {"type": "yarn", "displayName": "Jobs", "buttonName": "Jobs", "tooltip": "Jobs", "page": "/jobbrowser/"}, {"type": "importer", "displayName": "Importer", "buttonName": "Import", "tooltip": "Importer", "page": "/indexer/importer"}], "interpreter_names": ["hdfs", "tables", "yarn", "importer"]}, "home": {"name": "home", "displayName": "Home", "buttonName": "Documents", "interpreters": [], "page": "/home"}}, "main_button_action": {"name": "MySQL", "type": "mysql", "id": "mysql", "displayName": "MySQL", "buttonName": "Query", "tooltip": "Mysql Query", "optimizer": "off", "page": "/editor/?type=mysql", "is_sql": true, "is_batchable": true, "dialect": "mysql", "dialect_properties": {}}, "button_actions": [{"name": "editor", "displayName": "Editor", "buttonName": "Query", "interpreters": [{"name": "MySQL", "type": "mysql", "id": "mysql", "displayName": "MySQL", "buttonName": "Query", "tooltip": "Mysql Query", "optimizer": "off", "page": "/editor/?type=mysql", "is_sql": true, "is_batchable": true, "dialect": "mysql", "dialect_properties": {}}, {"name": "notebook", "type": "notebook", "displayName": "Notebook", "buttonName": "Notebook", "tooltip": "Notebook", "page": "/notebook", "is_sql": false, "dialect": "notebook"}], "default_limit": 5000, "interpreter_names": ["mysql", "notebook"], "page": "/editor/?type=mysql", "default_sql_interpreter": "mysql"}], "default_sql_interpreter": "mysql", "cluster_type": "direct", "has_computes": false, "hue_config": {"enable_sharing": true, "is_admin": true}, "clusters": [{"id": "default", "name": "default", "type": "direct", "credentials": {}}], "documents": {"types": ["directory", "gist", "query-mysql"]}, "status": 0} +### Get the filesystems -## File Browsing +Get the filesystems you can use and your home directory on each. -Hue's [File Browser](https://docs.gethue.com/user/browsing/#data) offer uploads, downloads, operations (create, delete, chmod...) and listing of data in HDFS (`hdfs://` or no prefix), S3 (`s3a://` prefix), ADLS (`adls://` or `abfs://` prefixes), Ozone (`ofs://` prefix) storages. +**Method and URL:** `GET /api/v1/storage/filesystems` -### Get Filesystems +```bash +curl "$HUE_URL/api/v1/storage/filesystems" -H "Authorization: Bearer $TOKEN" +``` -Get the filesystems details such as configured filesystems in Hue which user has access to and its home directories: +```json +[{"name": "hdfs", "user_home_directory": "/user/demo", "config": {"is_trash_enabled": true, "is_hdfs_superuser": false, "groups": [], "users": [], "superuser": "hdfs", "supergroup": "supergroup"}}, {"name": "s3a", "user_home_directory": "s3a://demo-gethue/user/demo", "config": {}}] +``` - curl -X GET https://demo.gethue.com/api/v1/storage/filesystems - - [{"file_system": "hdfs", "user_home_directory": "/user/demo"}, {"file_system": "s3a", "user_home_directory": "s3a://"}, {"file_system": "abfs", "user_home_directory": "abfs://"}, {"file_system": "ofs", "user_home_directory": "ofs://"}] +### List a directory -### List +**Method and URL:** `GET /api/v1/storage/list` -Here is how to list the content of a path, here a S3 bucket `s3a://demo-gethue`: +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | No | The directory to list. The default is `/`. | +| pagenum | integer | No | Page number. The default is 1. | +| pagesize | integer | No | Page size. The default is 30. | +| sortby | string | No | `type`, `name` (default), `atime`, `mtime`, `user`, `group` or `size`. | +| descending | boolean | No | Sort in descending order. The default is `false`. | +| filter | string | No | Keep only the names that contain this text. | - curl -X GET https://demo.gethue.com/api/v1/storage/view=s3a://demo-gethue +```bash +curl "$HUE_URL/api/v1/storage/list?path=s3a://demo-gethue/&pagesize=45&pagenum=1" \ + -H "Authorization: Bearer $TOKEN" +``` +```json +{ + "files": [ { - ........... - "files": [ - { - "humansize": "0\u00a0bytes", - "url": "/filebrowser/view=s3a%3A%2F%2Fdemo-hue", - "stats": { - "size": 0, - "aclBit": false, - "group": "", - "user": "", - "mtime": null, - "path": "s3a://demo-gethue", - "atime": null, - "mode": 16895 - }, - "name": "demo-hue", - "mtime": "", - "rwx": "drwxrwxrwx", - "path": "s3a://demo-gethue", - "is_sentry_managed": false, + "path": "s3a://demo-gethue/data", "type": "dir", - "mode": "40777" - }, - { - "humansize": "0\u00a0bytes", - "url": "/filebrowser/view=S3A%3A%2F%2F", - "stats": { + "rwx": "drwxrwxrwx", "size": 0, - "aclBit": false, - "group": "", "user": "", + "group": "", "mtime": null, - "path": "S3A://", "atime": null, - "mode": 16895 - }, - "name": ".", - "mtime": "", - "rwx": "drwxrwxrwx", - "path": "S3A://", - "is_sentry_managed": false, - "type": "dir", - "mode": "40777" - } - ], - ........... + "mode": 16895, + "aclBit": false } + ], + "page": {"page_number": 1, "page_size": 45, "total_pages": 1, "total_size": 1} +} +``` -Some of the parameters: - - pagesize=45 - - pagenum=1 - - filter= - - sortby=name - - descending=false - -E.g. `?pagesize=45&pagenum=1&filter=&sortby=name&descending=false` - -### Preview - -How to get the some of the file content and its stats/metadata. - -Example with a S3 file: - - curl -X GET https://demo.gethue.com/api/v1/storage/view=s3a://demo-gethue/data/web_logs/index_data.csv +**Errors:** `400` when the path is not a directory, `404` when the page does not exist, `403` when listing all buckets is not allowed. - { - "show_download_button": true, - "is_embeddable": false, - "editable": false, - "mtime": "October 31, 2016 03:34 PM", - "rwx": "-rw-rw-rw-", - "path": "s3a://demo-gethue/data/web_logs/index_data.csv", - "stats": { - "size": 6199593, - "aclBit": false, - ............... - "contents": "code,protocol,request,app,user_agent_major,region_code,country_code,id,city,subapp,latitude,method,client_ip, user_agent_family,bytes,referer,country_name,extension,url,os_major,longitude,device_family,record,user_agent,time,os_family,country_code3 - 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,8836e6ce-9a21-449f-a372-9e57641389b3,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:49 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)"" - ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:49Z,Other,SGP - 200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1,metastore,,00,SG,6ddf6e38-7b83-423c-8873-39842dca2dbb,Singapore,table,1.2931000000000097,GET,128.199.234.236,Other,1041,-,Singapore,,/metastore/table/default/sample_07,,103.85579999999999,Other,"demo.gethue.com:80 128.199.234.236 - - [04/May/2014:06:35:50 +0000] ""GET /metastore/table/default/sample_07 HTTP/1.1"" 200 1041 ""-"" ""Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)"" - ",Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org),2014-05-04T06:35:50Z,Other,SGP - ............... - } +### Get the stats of a path -Some of the parameters: -- offset=0 -- length=204800 -- compression=none -- mode=text +**Method and URL:** `GET /api/v1/storage/stat` -E.g. `?offset=0&length=204800&compression=none&mode=text` +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file or directory. | -### Download +```bash +curl "$HUE_URL/api/v1/storage/stat?path=/user/demo/weblogs.csv" -H "Authorization: Bearer $TOKEN" +``` -Specify a path of the file to download: +The response has the same fields as an entry of the directory listing. It returns `404` when the path does not exist. - curl -X GET https://demo.gethue.com/api/v1/storage/download=/user/hue/weblogs.csv +### Preview a file - curl -X GET https://demo.gethue.com/api/v1/storage/download=s3a://demo-gethue/data/web_logs/index_data.csv +Read a part of a text file. -- **download:** path of the file you want to download +**Method and URL:** `GET /api/v1/storage/display` -### Upload +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file to read. | +| offset | integer | No | First byte to read. The default is 0. | +| length | integer | No | Number of bytes to read. The default is the server chunk size. A value above the maximum chunk size returns `400`. | +| begin, end | integer | No | Alternative to `offset` and `length`. First byte (starting at 1) and last byte. | +| compression | string | No | The compression of the file. | +| mode | string | No | Only `text` is accepted. | +| encoding | string | No | Character encoding. The default is the server encoding. | -Upload a local file to a remote destination directory: +```bash +curl "$HUE_URL/api/v1/storage/display?path=s3a://demo-gethue/data/web_logs/index_data.csv&offset=0&length=204800" \ + -H "Authorization: Bearer $TOKEN" +``` - curl -X POST https://demo.gethue.com/api/v1/storage/upload/file?dest=s3a://demo-gethue/web_log_data/ --form hdfs_file=@README.md +```json +{"contents": "code,protocol,request\n200,HTTP/1.1,GET /metastore/table/default/sample_07 HTTP/1.1\n", "offset": 0, "length": 204800, "end": 76, "mode": "text"} +``` -- **dest:** Path of the directory for uploading a file. A new directory is created in the specified path if it does not exist. -- **hdfs_file:** relative or absolute path to a file. It should be read more like a `local_file`, this field is not related to HDFS. +**Errors:** `400` when the path is not a file or the range is invalid, `422` when the content cannot be decoded as text. -### Create Directory +### Download a file -Create a directory at a specific path: +**Method and URL:** `GET /api/v1/storage/download` - curl -X POST https://demo.gethue.com/api/v1/storage/mkdir - -- **name:** name of the directory you want to create -- **path:** path where you want to create the directory +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file to download. | +| disposition | string | No | Set to `inline` to display the file instead of downloading it, when the file type allows it. | -### Create File +```bash +curl -o weblogs.csv "$HUE_URL/api/v1/storage/download?path=/user/demo/weblogs.csv" \ + -H "Authorization: Bearer $TOKEN" +``` -Create a file at a specific path: +**Errors:** `400` when the path is not a file, `403` when you cannot read it or downloads are disabled, `404` when it does not exist. - curl -X POST https://demo.gethue.com/api/v1/storage/touch - -- **name:** name of the file you want to create -- **path:** path where you want to create the file +### Upload a file -### Rename +**Method and URL:** `POST /api/v1/storage/upload/file` -Rename a file or directory: +Send the file as `multipart/form-data`. - curl -X POST https://demo.gethue.com/api/v1/storage/rename - -- **src_path:** path of the file or directory you want to rename -- **dest_path:** path after renaming the selected file or directory +| Name | Type | Required | Description | +|------|------|----------|-------------| +| destination_path | query string | Yes | The destination of the file. | +| overwrite | query boolean | No | Replace an existing file. The default is `false`. | +| file | form file | Yes | The file to upload. | -### Move +```bash +curl -X POST "$HUE_URL/api/v1/storage/upload/file?destination_path=s3a://demo-gethue/web_log_data/&overwrite=false" \ + -H "Authorization: Bearer $TOKEN" \ + -F "file=@README.md" +``` -Move a file or directory to a destination path: +A successful upload returns `201 Created` and a `file_stats` object that describes the new file. Errors return `{"error": ""}` with status `400`, `401`, `404` or `500`. - curl -X POST https://demo.gethue.com/api/v1/storage/move - -- **src_path:** path of the file or directory that you want to move -- **dest_path:** path at which you want to move the selected file or directory +### Create a directory -### Copy +**Method and URL:** `POST /api/v1/storage/create/directory` -Copy a file or directory to a destination path: +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The parent directory. | +| name | string | Yes | The name of the new directory. It cannot contain `/` or `#`. | - curl -X POST https://demo.gethue.com/api/v1/storage/copy - -- **src_path:** path of the file or directory you want to copy -- **dest_path:** path at which you want to save the copy of the selected file or directory +```bash +curl -X POST "$HUE_URL/api/v1/storage/create/directory" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo" -d "name=reports" +``` -**Note:** On the **Apache Ozone** filesystem, the copy operation returns a string of skipped files if their size is greater than the configured chunk size. +It returns `201 Created`. It returns `409` when the directory exists and `400` when a parameter is missing or invalid. -### Get Content Summary +### Create a file -Fetch the content summary for a specific file on **HDFS** or **Apache Ozone**: +**Method and URL:** `POST /api/v1/storage/create/file` - curl -X GET https://demo.gethue.com/api/v1/storage/content_summary=/user/hue/weblogs.csv +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The parent directory. | +| name | string | Yes | The name of the new file. It cannot contain `/`. | - curl -X GET https://demo.gethue.com/api/v1/storage/content_summary=ofs://ozone1/testvolume/testbucket/testfile.csv +```bash +curl -X POST "$HUE_URL/api/v1/storage/create/file" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo" -d "name=notes.txt" +``` -### Delete +It returns `201 Created`. It returns `409` when the file exists and `400` when a parameter is missing or invalid. -Delete a file or directory: +### Save a file - curl -X POST https://demo.gethue.com/api/v1/storage/rmtree - -- **path:** path of the file or directory you want to delete -- **skip_trash:** Boolean value to indicate whether to move the deleted file or directory to the trash directory. Specify `False` to move the file to the trash. +Write the content of a file. It creates the file when the path does not exist. -**Note:** Currently, the `skip_trash` field is only supported on HDFS. +**Method and URL:** `POST /api/v1/storage/save` -### Set Replication +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file to write. | +| contents | string | Yes | The new content. | +| encoding | string | Yes | The character encoding, for example `utf-8`. | -Set the replication factor for a file on **HDFS**: +```bash +curl -X POST "$HUE_URL/api/v1/storage/save" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/notes.txt" -d "encoding=utf-8" --data-urlencode "contents=Hello" +``` - curl -X POST https://demo.gethue.com/api/v1/storage/set_replication - -- **src_path:** path of the file for setting the replication factor -- **replication_factor:** numerical value for setting the replication factor for the file +It returns `200 OK`. -### Restore Trash +### Rename -Restore a specific file or directory from trash on **HDFS**: +**Method and URL:** `POST /api/v1/storage/rename` - curl -X POST https://demo.gethue.com/api/v1/storage/trash/restore - -- **path:** path of the file or directory that you want to restore from the trash directory +Send a JSON body. -### Purge Trash +| Name | Type | Required | Description | +|------|------|----------|-------------| +| source_path | string | Yes | The file or directory to rename. | +| destination_path | string | Yes | The new path. It can be relative to the parent of the source. | -Purge the trash directory on **HDFS**: +```bash +curl -X POST "$HUE_URL/api/v1/storage/rename" \ + -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ + -d '{"source_path": "/user/demo/a.csv", "destination_path": "/user/demo/b.csv"}' +``` - curl -X POST https://demo.gethue.com/api/v1/storage/trash/purge +It returns `200 OK`, or `400` with the validation errors. -## Importer +### Move and copy -The File Import API provides endpoints for uploading, analyzing, and previewing files that can be imported into various SQL engines. This API simplifies the process of creating database tables from files like CSV, TSV, and Excel spreadsheets. +**Method and URL:** -### Overview +- `POST /api/v1/storage/move` +- `POST /api/v1/storage/copy` -The File Import API allows you to: +| Name | Type | Required | Description | +|------|------|----------|-------------| +| source_path | string | Yes | The file or directory. | +| destination_path | string | Yes | The destination directory. It must already exist. | -- Upload files from your local system -- Analyze file metadata to determine format and characteristics -- Check whether files have headers -- Preview file content with data types -- Get SQL type mappings for different SQL dialects +```bash +curl -X POST "$HUE_URL/api/v1/storage/copy" \ + -H "Authorization: Bearer $TOKEN" \ + -d "source_path=/user/demo/a.csv" -d "destination_path=/user/demo/archive" +``` -### Typical Import Workflow +It returns `200 OK`. It returns `409` when a file with the same name already exists in the destination directory and `400` when a parameter is missing or invalid, or when the destination is not a directory. On Ozone, a copy can return `500` with `{"skipped_files": ...}` for files larger than the configured chunk size. -A typical workflow for importing a file into a database table involves these steps: +### Delete -1. **Upload the file** using the `/api/v1/importer/upload/file/` endpoint -2. **Detect file metadata** using the `/api/v1/importer/file/guess_metadata/` endpoint -3. **Determine if the file has a header** using the `/api/v1/importer/file/guess_header/` endpoint -4. **Preview the file** with column type detection using the `/api/v1/importer/file/preview/` endpoint -5. Use the preview data to create a table in your SQL engine of choice +**Method and URL:** `POST /api/v1/storage/delete` -### Upload a Local File +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file or directory. | +| skip_trash | boolean | No | Delete permanently instead of moving to the trash. The default is `false`. Only HDFS supports the trash. | -Upload a file from your local system to the Hue server. +```bash +curl -X POST "$HUE_URL/api/v1/storage/delete" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/a.csv" -d "skip_trash=false" +``` -**Endpoint:** `/api/v1/importer/upload/file/` +It returns `200 OK`. -**Method:** `POST` +### Change the owner or permissions -**Content Type:** `multipart/form-data` +**Method and URL:** -**Request Parameters:** +- `POST /api/v1/storage/chown` +- `POST /api/v1/storage/chmod` | Name | Type | Required | Description | |------|------|----------|-------------| -| file | File | Yes | The file to upload (csv, tsv, excel) | +| path | string | Yes | The file or directory. | +| user | string | `chown` | The new owner. | +| group | string | `chown` | The new group. | +| recursive | boolean | No | `chown` only. Apply to the content of a directory. | +| permission | JSON string | `chmod` | The permission flags. | -**Example using cURL:** +The `permission` object has the boolean keys `user_read`, `user_write`, `user_execute`, `group_read`, `group_write`, `group_execute`, `other_read`, `other_write`, `other_execute`, `sticky` and `recursive`. ```bash -curl -X POST \ - -H "Authorization: Bearer " \ - -F "file=@/path/to/sales_data.csv" \ - https://demo.gethue.com/api/v1/importer/upload/file/ +curl -X POST "$HUE_URL/api/v1/storage/chmod" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/a.csv" \ + -d 'permission={"user_read":true,"user_write":true,"group_read":true,"other_read":true}' ``` -**Response:** +It returns `200 OK`. -```json -{ - "file_path": "/tmp/username_abc123_sales_data.csv" -} -``` +### Get a content summary -**Status Codes:** +Get the size and the number of files of a path on HDFS or Ozone. -- `201 Created` - File was uploaded successfully -- `400 Bad Request` - Invalid file format or size -- `500 Internal Server Error` - Server-side error +**Method and URL:** `GET /api/v1/storage/content_summary` -**Restrictions:** -- Maximum file size is determined by the configuration `IMPORTER.MAX_LOCAL_FILE_SIZE_UPLOAD_LIMIT` -- Certain file extensions may be restricted based on `IMPORTER.RESTRICT_LOCAL_FILE_EXTENSIONS` +| Name | Type | Required | Description | +|------|------|----------|-------------| +| path | string | Yes | The file or directory. | -### Guess File Metadata +```bash +curl "$HUE_URL/api/v1/storage/content_summary?path=/user/demo/weblogs.csv" \ + -H "Authorization: Bearer $TOKEN" +``` -Analyze a file to determine its type and metadata properties such as delimiters for CSV files or sheet names for Excel files. +It returns `400` when the path is missing, `404` when it does not exist and `500` when the summary fails. -**Endpoint:** `/api/v1/importer/file/guess_metadata/` +### Set the replication factor -**Method:** `GET` +Set the replication factor of a file on HDFS. -**Request Parameters:** +**Method and URL:** `POST /api/v1/storage/replication` | Name | Type | Required | Description | |------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to analyze | -| import_type | String | Yes | Type of import, either `local` or `remote` | - -**Example using cURL:** +| path | string | Yes | The file. | +| replication_factor | integer | Yes | The new replication factor. | ```bash -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_metadata/?file_path=/tmp/username_abc123_sales_data.csv&import_type=local" +curl -X POST "$HUE_URL/api/v1/storage/replication" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/weblogs.csv" -d "replication_factor=2" ``` -**Response Examples:** +### Trash -For CSV files: -```json -{ - "type": "csv", - "field_separator": ",", - "quote_char": "\"", - "record_separator": "\n" -} +The trash is only available on HDFS. + +| Purpose | Method and URL | Parameters | +|---------|----------------|------------| +| Get the trash path of a path | `GET /api/v1/storage/trash/path` | `path` | +| Restore a path from the trash | `POST /api/v1/storage/trash/restore` | `path` | +| Empty your trash | `POST /api/v1/storage/trash/purge` | None | + +```bash +curl -X POST "$HUE_URL/api/v1/storage/trash/restore" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/.Trash/Current/user/demo/a.csv" ``` -For Excel files: -```json -{ - "type": "excel", - "sheet_names": ["Sales 2024", "Sales 2025", "Analytics"] -} +### Bulk operations + +These endpoints apply an operation to several paths. Repeat the `path` parameter (or `source_path` for move and copy) once for each item. + +| Operation | Method and URL | +|-----------|----------------| +| Move | `POST /api/v1/storage/move/bulk` | +| Copy | `POST /api/v1/storage/copy/bulk` | +| Delete | `POST /api/v1/storage/delete/bulk` | +| Restore from the trash | `POST /api/v1/storage/trash/restore/bulk` | +| Change the owner | `POST /api/v1/storage/chown/bulk` | +| Change the permissions | `POST /api/v1/storage/chmod/bulk` | + +```bash +curl -X POST "$HUE_URL/api/v1/storage/delete/bulk" \ + -H "Authorization: Bearer $TOKEN" \ + -d "path=/user/demo/a.csv" -d "path=/user/demo/b.csv" ``` -### Guess File Header +It returns `200 OK` when all operations succeed. Otherwise, it returns `500` with an object that maps each failed path to its error. -Analyze a file to determine if it has a header row. +## Importer -**Endpoint:** `/api/v1/importer/file/guess_header/` +Use the Importer endpoints to upload a file, detect its format and preview it before you create a table. Read the [File Import API](/developer/api/rest/importer/) for the parameters, responses and a full workflow. -**Method:** `GET` +| Purpose | Method and URL | +|---------|----------------| +| Upload a local file | `POST /api/v1/importer/upload/file/` | +| Detect the file format | `GET /api/v1/importer/file/guess_metadata/` | +| Detect a header row | `GET /api/v1/importer/file/guess_header/` | +| Preview a file | `GET /api/v1/importer/file/preview/` | +| List the SQL types of a dialect | `GET /api/v1/importer/sql_type_mapping/` | -**Request Parameters:** +## Connectors -| Name | Type | Required | Description | -|------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to analyze | -| file_type | String | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | -| import_type | String | Yes | Type of import, either `local` or `remote` | -| sheet_name | String | No | Sheet name (required for Excel files) | +[Connectors](/administrator/configuration/connectors/) describe the interpreters, browsers and catalogs of the server. Creating, updating, deleting and testing a connector requires admin privileges. -**Example using cURL:** +### List the instances + +**Method and URL:** `GET /api/v1/connector/instances` ```bash -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_header/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local" +curl "$HUE_URL/api/v1/connector/instances" -H "Authorization: Bearer $TOKEN" ``` -**Response:** - ```json -{ - "has_header": true -} +{"connectors": [{"category": "editor", "category_name": "Editor", "description": "", "values": []}, {"category": "browsers", "category_name": "Browsers", "description": "", "values": []}]} ``` -### Preview File +The categories are `editor`, `browsers`, `catalogs`, `optimizers`, `schedulers` and `plugins`. -Generate a preview of a file's content with column type mapping for creating SQL tables. +### List the types -**Endpoint:** `/api/v1/importer/file/preview/` +**Method and URL:** `GET /api/v1/connector/types` + +```bash +curl "$HUE_URL/api/v1/connector/types" -H "Authorization: Bearer $TOKEN" +``` -**Method:** `GET` +The response has a `connectors` list grouped by category, and a `categories` list. Each connector type has a `dialect`, an `interface`, a `nice_name` and its default `settings`. -**Request Parameters:** +### Get a template for a new instance -| Name | Type | Required | Description | -|------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to preview | -| file_type | String | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | -| import_type | String | Yes | Type of import (`local` or `remote`) | -| sql_dialect | String | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | -| has_header | Boolean | Yes | Whether the file has a header row | -| sheet_name | String | No | Sheet name (required for Excel files) | -| field_separator | String | No | Field separator character (defaults to `,` for CSV, `\t` for TSV, required for `delimiter_format`) | -| quote_char | String | No | Quote character (defaults to `"`) | -| record_separator | String | No | Record separator character (defaults to `\n`) | +Pick a `dialect` and `interface` from the list of types. -**Example using cURL:** +**Method and URL:** `POST /api/v1/connector/instance/new//` ```bash -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local&sql_dialect=hive&has_header=true" - -# For a custom pipe-delimited file using delimiter_format -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_pipe_data.txt&file_type=delimiter_format&import_type=local&sql_dialect=hive&has_header=true&field_separator=|"e_char=\"&record_separator=\n" +curl -X POST "$HUE_URL/api/v1/connector/instance/new/hive/hiveserver2" \ + -H "Authorization: Bearer $TOKEN" ``` -**About `delimiter_format` File Type:** +The response is `{"connector": {...}}`. Edit it and send it to the update call. -The `delimiter_format` file type should be used for custom delimited files that don't follow standard CSV or TSV formats. When using this file type: -- `field_separator` is required and must be explicitly specified -- `quote_char` and `record_separator` should be provided for proper parsing -- Values from the `guess_metadata` response should be passed to ensure consistent parsing +### Get an instance -**Parameter Validation Notes:** -- For Excel files, `sheet_name` is required -- For standard formats (CSV/TSV), appropriate defaults are applied -- For `delimiter_format`, always specify the required parameters -- It's recommended to always pass the `record_separator` from the `guess_metadata` response +**Method and URL:** `GET /api/v1/connector/instance/get/` -**Response:** +```bash +curl "$HUE_URL/api/v1/connector/instance/get/41" -H "Authorization: Bearer $TOKEN" +``` ```json -{ - "type": "csv", - "columns": [ - { - "name": "transaction_id", - "type": "INT" - }, - { - "name": "product_name", - "type": "STRING" - }, - { - "name": "price", - "type": "DOUBLE" - } - ], - "preview_data": [ - ["1001", "Laptop XPS 13", "1299.99"], - ["1002", "Wireless Headphones", "149.99"], - ["1003", "Office Chair", "249.50"] - ] -} +{"id": 41, "name": "Hive Docker Local", "description": "Recommended", "dialect": "hive", "settings": [{"name": "server_host", "value": "localhost"}]} ``` -### Get SQL Type Mapping +### Create or update an instance -Get the list of unique SQL data types supported by a specific SQL dialect. - -**Endpoint:** `/api/v1/importer/sql_type_mapping/` - -**Method:** `GET` - -**Request Parameters:** +**Method and URL:** `POST /api/v1/connector/instance/update` | Name | Type | Required | Description | |------|------|----------|-------------| -| sql_dialect | String | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | - -**Example using cURL:** +| connector | JSON string | Yes | The connector. Without an `id`, Hue creates a new instance. With an `id`, it updates that instance. It needs `nice_name`, `description` and `settings`. A new instance also needs `dialect` and `interface`. | ```bash -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/sql_type_mapping/?sql_dialect=hive" +curl -X POST "$HUE_URL/api/v1/connector/instance/update" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'connector={"nice_name":"Hive Docker Local","description":"Recommended","dialect":"hive","interface":"hiveserver2","settings":[{"name":"server_host","value":"localhost"},{"name":"server_port","value":10000}]}' ``` -**Response:** - ```json -[ - "ARRAY", - "BIGINT", - "BINARY", - "BOOLEAN", - "DATE", - "DECIMAL", - "DOUBLE", - "FLOAT", - "INT", - "INTERVAL DAY TO SECOND", - "SMALLINT", - "STRING", - "STRUCT", - "TIMESTAMP", - "TINYINT" -] -``` - -### Complete Workflow Example - -Here's an example workflow that combines all the APIs to import a CSV file into a Hive table: - -1. **Upload the file** -2. **Detect file metadata** -3. **Check for header row** -4. **Preview the file with column type detection** -5. **Generate SQL CREATE TABLE statement** - -For the full code example and best practices, refer to the [File Import documentation](/developer/api/rest/importer/). - -## Connectors - -### List - -Get the list of configured [connectors](/administrator/configuration/connectors/): - - curl -X GET https://demo.gethue.com/api/v1/connector/instances +{"connector": {"id": 41, "name": 41, "nice_name": "Hive Docker Local"}, "saved_as": true} +``` - {"connectors": [{"category": "editor", "category_name": "Editor", "description": "", "values": []}, {"category": "browsers", "category_name": "Browsers", "description": "", "values": []}, {"category": "catalogs", "category_name": "Catalogs", "description": "", "values": []}, {"category": "optimizers", "category_name": "Optimizers", "description": "", "values": []}, {"category": "schedulers", "category_name": "Schedulers", "description": "", "values": []}, {"category": "plugins", "category_name": "Plugins", "description": "", "values": []}]} +### Delete an instance -### Types +**Method and URL:** `POST /api/v1/connector/instance/delete` - curl -X GET https://demo.gethue.com/api/v1/connector/types +```bash +curl -X POST "$HUE_URL/api/v1/connector/instance/delete" \ + -H "Authorization: Bearer $TOKEN" \ + -d 'connector={"id": "41"}' +``` - { "connectors": [ { "category": "editor", "category_name": "Editor", "description": "", "values": [ { "dialect": "hive", "nice_name": "Hive", "description": "Recommended", "category": "editor", "interface": "hiveserver2", "settings": [ { "name": "server_host", "value": "localhost" }, { "name": "server_port", "value": 10000 }, { "name": "is_llap", "value": false }, { "name": "use_sasl", "value": true } ], "properties": { "is_sql": true, "sql_identifier_quote": "`", "sql_identifier_comment_single": "--", "has_catalog": false, "has_database": true, "has_table": true, "has_live_queries": false, "has_optimizer_risks": true, "has_optimizer_values": true, "has_auto_limit": false, "has_reference_language": true, "has_reference_functions": true, "has_use_statement": true } }, - ........... - { "category": "browsers", "category_name": "Browsers", "description": "", "values": [ { "nice_name": "HDFS", "dialect": "hdfs", "interface": "rest", "settings": [ { "name": "server_url", "value": "http://localhost:50070/webhdfs/v1" }, { "name": "default_fs", "value": "fs_defaultfs=hdfs://localhost:8020" } ], "category": "browsers", "description": "", "properties": {} }, - ........... - { "nice_name": "S3", "dialect": "s3", "settings": [], "category": "browsers", "description": "", "properties": {} }, { "nice_name": "ADLS", "dialect": "adls-v1", "settings": [], "category": "browsers", "description": "", "properties": {} } ] }, { "category": "catalogs", "category_name": "Catalogs", "description": "", "values": [ { "nice_name": "Hive Metastore", "dialect": "hms", "interface": "hiveserver2", "settings": [ { "name": "server_host", "value": "" }, { "name": "server_port", "value": "" } ], "category": "catalogs", "description": "", "properties": {} }, { "nice_name": "Atlas", "dialect": "atlas", "interface": "rest", "settings": [], "category": "catalogs", "description": "", "properties": {} }, - ........... - ] }, { "category": "optimizers", "category_name": "Optimizers", "description": "", "values": [ { "nice_name": "Optimizer", "dialect": "optimizer", "settings": [], "category": "optimizers", "description": "", "properties": {} } ] }, { "category": "schedulers", "category_name": "Schedulers", "description": "", "values": [ { "nice_name": "Oozie", "dialect": "oozie", "settings": [], "category": "schedulers", "description": "", "properties": {} }, - ........... - ] }, { "category": "plugins", "category_name": "Plugins", "description": "", "values": [] } ], "categories": [ { "name": "Editor", "type": "editor", "description": "" }, { "name": "Browsers", "type": "browsers", "description": "" }, { "name": "Catalogs", "type": "catalogs", "description": "" }, { "name": "Optimizers", "type": "optimizers", "description": "" }, { "name": "Schedulers", "type": "schedulers", "description": "" }, { "name": "Plugins", "type": "plugins", "description": "" } ] } +### Test an instance -### Create +Check that the connection works. -First step is to get the config of the connector we want to instantiate. In input we pick a type of connector from the list of above types by specifying its `dialect` and `interface` names. +**Method and URL:** `POST /api/v1/connector/instance/test` - curl -X POST https://demo.gethue.com/api/v1/connector/instance/new// +The `connector` parameter is the same JSON object as for the update call. -And get back a template that we send to the /update call: +```bash +curl -X POST "$HUE_URL/api/v1/connector/instance/test" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'connector={"nice_name":"Hive Docker Local","dialect":"hive","interface":"hiveserver2","settings":[{"name":"server_host","value":"localhost"}]}' +``` - curl -X POST https://demo.gethue.com/api/v1/connector/instance/new/hive/sqlalchemy -d 'connector={"nice_name":"Hive Docker Local","name":"41","dialect":"hive","interface":"hiveserver2","settings":[{"name":"server_host","value":"localhost"},{"name":"server_port","value":10000},{"name":"is_llap","value":false},{"name":"use_sasl","value":"true"}],"category":"editor","description":"Recommended","dialect_properties":{"is_sql":true,"sql_identifier_quote":"`","sql_identifier_comment_single":"--","has_catalog":false,"has_database":true,"has_table":true,"has_live_queries":false,"has_optimizer_risks":true,"has_optimizer_values":true,"has_auto_limit":false,"has_reference_language":true,"has_reference_functions":true,"has_use_statement":true}}' +```json +{"warnings": "", "hasWarnings": false} +``` -### Get +### Install the examples - curl -X GET https://demo.gethue.com/api/v1/connector/instance/get/ +Install or update the connector examples. -### Update +**Method and URL:** `POST /api/v1/connector/examples/install` -This is the same as creating a new connector instance, but as we provide the `id` we will update the existing instance: +```bash +curl -X POST "$HUE_URL/api/v1/connector/examples/install" -H "Authorization: Bearer $TOKEN" +``` - curl -X POST https://demo.gethue.com/api/v1/connector/instance/update -d 'connector={"nice_name":"Hive Docker Local","name":"41","dialect":"hive","interface":"hiveserver2","settings":[{"name":"server_host","value":"localhost"},{"name":"server_port","value":10000},{"name":"is_llap","value":false},{"name":"use_sasl","value":"true"}],"id":"41","category":"editor","description":"Recommended","dialect_properties":{"is_sql":true,"sql_identifier_quote":"`","sql_identifier_comment_single":"--","has_catalog":false,"has_database":true,"has_table":true,"has_live_queries":false,"has_optimizer_risks":true,"has_optimizer_values":true,"has_auto_limit":false,"has_reference_language":true,"has_reference_functions":true,"has_use_statement":true}}' +```json +{"status": 0, "message": "Added connectors: ..."} +``` -### Delete +## Users - curl -X POST https://demo.gethue.com/api/v1/connector/instance/delete -d 'connector={"id": "1"}' +### List the users -### Test +Requires admin privileges. -Check if the connectivity is healthy: +**Method and URL:** `GET /api/v1/iam/get_users` - curl -X POST https://demo.gethue.com/api/v1/connector/instance/test/ -d 'connector={"nice_name":"Hive Docker Local","name":"41","dialect":"hive","interface":"hiveserver2","settings":[{"name":"server_host","value":"localhost"},{"name":"server_port","value":10000},{"name":"is_llap","value":false},{"name":"use_sasl","value":"true"}],"id":"41","category":"editor","description":"Recommended","dialect_properties":{"is_sql":true,"sql_identifier_quote":"`","sql_identifier_comment_single":"--","has_catalog":false,"has_database":true,"has_table":true,"has_live_queries":false,"has_optimizer_risks":true,"has_optimizer_values":true,"has_auto_limit":false,"has_reference_language":true,"has_reference_functions":true,"has_use_statement":true}}' +| Name | Type | Required | Description | +|------|------|----------|-------------| +| username | string | No | Filter on a username. | +| groups | string | No | Filter on a group. Repeat it to match several groups. | +| is_active | boolean | No | Set to `true` to keep only the active users. | -### Examples +```bash +curl "$HUE_URL/api/v1/iam/get_users?username=demo&is_active=true" \ + -H "Authorization: Bearer $TOKEN" +``` -Install or update the connector examples: +### Get users by id - curl -X POST https://demo.gethue.com/api/v1/connector/examples/install/ +Requires admin privileges, or sharing enabled on the server. -## IAM +**Method and URL:** `GET /api/v1/iam/users` -### Get users +| Name | Type | Required | Description | +|------|------|----------|-------------| +| userids | JSON list | Yes | Up to 100 user ids. | -Get user records in Hue. Requires **admin privileges**. +```bash +curl -G "$HUE_URL/api/v1/iam/users" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'userids=[1100714,1100715]' +``` - curl -X GET https://demo.gethue.com/api/v1/iam/get_users +```json +{"users": [{"id": 1100714, "username": "demo", "first_name": "", "last_name": "", "email": "", "last_login": "2021-10-06T01:36:49.663", "editURL": "/useradmin/users/edit/demo"}]} +``` -Optional GET params: -- **username:** filter by username -- **groups:** filter by specific group -- **is_active:** filter by active status +### Search users and groups -E.g. `?username=demo&groups=default&is_active=true` +**Method and URL:** `GET /api/v1/iam/users/autocomplete` -Search user records by list of user IDs. Requires **admin privileges**. +| Name | Type | Required | Description | +|------|------|----------|-------------| +| filter | string | No | Search text. | +| count | integer | No | Maximum number of results. The default is 100. | +| extend_user | boolean | No | Set to `true` to include the groups of each user. | +| include_myself | boolean | No | Include your own user in the results. | +| only_mygroups | boolean | No | Admins only. Return only your groups. | - curl -X GET https://demo.gethue.com/api/v1/iam/users?userids=[1100714,1100715] - - {"users": [{"id": 1100714,"username": "demo","first_name": "","last_name": "","email": "","last_login": "2021-10-06T01:36:49.663","editURL": "/useradmin/users/edit/demo"},{"id": 1100715,"username": "hue","first_name": "","last_name": "","email": "","last_login": "2021-08-11T07:15:48.793","editURL": "/useradmin/users/edit/hue"}]} +```bash +curl "$HUE_URL/api/v1/iam/users/autocomplete?filter=de&count=10" \ + -H "Authorization: Bearer $TOKEN" +``` -User list_for_autocomplete API: +Non-admin users only see the users of their groups, and only when sharing is enabled. - curl -X GET https://demo.gethue.com/api/v1/iam/users/autocomplete - -Optional GET params: -- **extend_user:** true or false (info about each user's groups) -- **filter:** search term -- **count:** Number of records (default is 100) +## Data catalog -## Data Catalog +Search for documents, tables and files. -The [metadata API](https://github.com/cloudera/hue/tree/master/desktop/libs/metadata) is powering the external [Catalog integrations](/user/browsing/#data-catalogs). +**Method and URL:** `POST /api/v1/metadata/search/entities_interactive/` -### Searching for entities - - curl -X POST https://demo.gethue.com/api/v1/metadata/search/entities_interactive/ -d 'query_s="*sample"&sources=["documents", "sql", "hdfs", "s3"]' +| Name | Type | Required | Description | +|------|------|----------|-------------| +| query_s | JSON string | Yes | The search text, for example `"*sample"` (with the quotes). | +| sources | JSON list | Yes | Where to search, for example `["documents", "sql"]`. | +| limit | integer | No | Maximum number of results. The default is 25. | +| field_facets | JSON list | No | The facets to return, for example `["type", "owner"]`. | +| interface | string | No | The catalog to use. The default is the configured catalog. | -Some of the parameters: -- **query_s:** search term -- **sources:** sources to search from `["documents", "sql", "hdfs", "s3"]` -- **field_facets:** `['type', 'owner', 'tags', 'lastModified']` -- **limit:** 10 +```bash +curl -X POST "$HUE_URL/api/v1/metadata/search/entities_interactive/" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'query_s="*sample"' \ + --data-urlencode 'sources=["documents"]' +``` -Searching for entities with the `dummy` catalog: +```json +{"status": 0, "count": 1, "results": [{"hue_name": "sample", "hue_description": "", "link": "/editor?editor=2008", "doc_type": "query-hive", "last_modified": "2021-05-20T08:09:30.360", "owner": "demo", "type": "HUE", "uuid": "5b48c678-1224-4863-b523-3baab82402a7", "parentUuid": "d1a5a0a9-0f2b-4d3f-a8a1-4ac1d7b0a7d1", "originalName": "sample"}]} +``` - curl -X POST https://demo.gethue.com/api/v1/metadata/search/entities_interactive/ -d 'query_s="*sample"&interface="dummy"' +When `sources` contains `documents`, Hue searches its own documents. Otherwise, it queries the configured catalog and returns `{"status": 1, "message": "Navigator not enabled"}` if there is none. diff --git a/docs/docs-site/content/developer/api/rest/importer.md b/docs/docs-site/content/developer/api/rest/importer.md index 3b41b6ff757..47c2b053fce 100644 --- a/docs/docs-site/content/developer/api/rest/importer.md +++ b/docs/docs-site/content/developer/api/rest/importer.md @@ -5,12 +5,16 @@ draft: false weight: 4 --- -The File Import API provides endpoints for uploading, analyzing, and previewing files that can be imported into various SQL engines. This API simplifies the process of creating database tables from files like CSV, TSV, and Excel spreadsheets. +Use the File Import API to upload, analyze and preview files before you import them into a SQL engine. It supports CSV, TSV, other delimited files and Excel spreadsheets. -> **Note**: All API endpoints require authentication. For endpoints that work with remote files (`import_type=remote`), the API uses the file system permissions associated with the authenticated user's session. +All endpoints require [authentication](/developer/api/rest/#authentication). For remote files (`import_type=remote`), the API uses the file system permissions of the authenticated user. + +The examples use `hue.example.com` as the server. Replace `` with the `access` token of your session. ## Overview +The importer is turned off by default. An administrator turns it on with `is_enabled=true` in the `[desktop] [[importer]]` section of `hue.ini`. The `get_config` response reports the setting in `importer.is_enabled`. + The File Import API allows you to: - Upload files from your local system @@ -19,7 +23,7 @@ The File Import API allows you to: - Preview file content with data types - Get SQL type mappings for different SQL dialects -## Typical Import Workflow +## Typical import workflow A typical workflow for importing a file into a database table involves these steps: @@ -29,42 +33,42 @@ A typical workflow for importing a file into a database table involves these ste 4. **Preview the file** with column type detection using the `/api/v1/importer/file/preview/` endpoint 5. Use the preview data to create a table in your SQL engine of choice -## Upload a Local File +## Upload a local file Upload a file from your local system to the Hue server. -**Endpoint:** `/api/v1/importer/upload/file/` +**Method and URL:** `POST /api/v1/importer/upload/file/` -**Method:** `POST` +**Content type:** `multipart/form-data` -**Content Type:** `multipart/form-data` - -**Request Parameters:** +**Request parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| -| file | File | Yes | The file to upload (csv, tsv, excel) | +| file | file | Yes | The file to upload (CSV, TSV or Excel). | + +The server saves the file in its temporary directory. Use the returned `file_path` with `import_type=local` in the other calls. Only the user who uploaded the file can use that path. -**Example using cURL:** +**Example request (curl):** ```bash curl -X POST \ - -H "Authorization: Bearer " \ + -H "Authorization: Bearer " \ -F "file=@/path/to/sales_data.csv" \ - https://demo.gethue.com/api/v1/importer/upload/file/ + https://hue.example.com/api/v1/importer/upload/file/ ``` -**Example using JavaScript:** +**Example request (JavaScript):** ```javascript // Using fetch API const formData = new FormData(); formData.append('file', fileInputElement.files[0]); -fetch('https://demo.gethue.com/api/v1/importer/upload/file/', { +fetch('https://hue.example.com/api/v1/importer/upload/file/', { method: 'POST', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' }, body: formData }) @@ -73,7 +77,7 @@ fetch('https://demo.gethue.com/api/v1/importer/upload/file/', { .catch(error => console.error('Error:', error)); ``` -**Response:** +**Example response:** ```json { @@ -81,53 +85,52 @@ fetch('https://demo.gethue.com/api/v1/importer/upload/file/', { } ``` -**Status Codes:** +**Status codes:** - `201 Created` - File was uploaded successfully - `400 Bad Request` - Invalid file format or size - `500 Internal Server Error` - Server-side error **Restrictions:** -- Maximum file size is determined by the configuration `IMPORTER.MAX_LOCAL_FILE_SIZE_UPLOAD_LIMIT` -- Certain file extensions may be restricted based on `IMPORTER.RESTRICT_LOCAL_FILE_EXTENSIONS` -## Guess File Metadata +- The maximum file size is set by `max_local_file_size_upload_limit` in the `[desktop] [[importer]]` section of `hue.ini`. The default is 150 MiB. +- An administrator can block file extensions with `restrict_local_file_extensions` in the same section. -Analyze a file to determine its type and metadata properties such as delimiters for CSV files or sheet names for Excel files. +## Guess file metadata -**Endpoint:** `/api/v1/importer/file/guess_metadata/` +Analyze a file to determine its type and metadata properties such as delimiters for CSV files or sheet names for Excel files. -**Method:** `GET` +**Method and URL:** `GET /api/v1/importer/file/guess_metadata/` -**Request Parameters:** +**Request parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to analyze | -| import_type | String | Yes | Type of import, either `local` or `remote` | +| file_path | string | Yes | Full path to the file to analyze | +| import_type | string | Yes | Type of import, either `local` or `remote` | -**Example using cURL:** +**Example request (curl):** ```bash # For a local file uploaded previously curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_metadata/?file_path=/tmp/username_abc123_sales_data.csv&import_type=local" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/guess_metadata/?file_path=/tmp/username_abc123_sales_data.csv&import_type=local" # For a remote file on HDFS curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_metadata/?file_path=/user/hue/data/sales_data.csv&import_type=remote" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/guess_metadata/?file_path=/user/hue/data/sales_data.csv&import_type=remote" ``` -**Example using JavaScript:** +**Example request (JavaScript):** ```javascript // Using fetch API for a local file -fetch('https://demo.gethue.com/api/v1/importer/file/guess_metadata/?file_path=/tmp/username_abc123_sales_data.csv&import_type=local', { +fetch('https://hue.example.com/api/v1/importer/file/guess_metadata/?file_path=/tmp/username_abc123_sales_data.csv&import_type=local', { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }) .then(response => response.json()) @@ -135,7 +138,7 @@ fetch('https://demo.gethue.com/api/v1/importer/file/guess_metadata/?file_path=/t .catch(error => console.error('Error:', error)); ``` -**Response Examples:** +**Example responses:** For CSV files: ```json @@ -143,7 +146,7 @@ For CSV files: "type": "csv", "field_separator": ",", "quote_char": "\"", - "record_separator": "\n" + "record_separator": "\r\n" } ``` @@ -153,7 +156,7 @@ For TSV files: "type": "tsv", "field_separator": "\t", "quote_char": "\"", - "record_separator": "\n" + "record_separator": "\r\n" } ``` @@ -165,44 +168,42 @@ For Excel files: } ``` -**Status Codes:** +**Status codes:** - `200 OK` - Metadata detected successfully - `400 Bad Request` - File not found or invalid parameters - `500 Internal Server Error` - Server-side error during detection -## Guess File Header +## Guess file header Analyze a file to determine if it has a header row. This API uses heuristics to detect if the first row appears to contain column names rather than data. -**Endpoint:** `/api/v1/importer/file/guess_header/` - -**Method:** `GET` +**Method and URL:** `GET /api/v1/importer/file/guess_header/` -**Request Parameters:** +**Request parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to analyze | -| file_type | String | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | -| import_type | String | Yes | Type of import, either `local` or `remote` | -| sheet_name | String | No | Sheet name (required for Excel files) | +| file_path | string | Yes | Full path to the file to analyze | +| file_type | string | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | +| import_type | string | Yes | Type of import, either `local` or `remote` | +| sheet_name | string | No | Sheet name (required for Excel files) | -**Example using cURL:** +**Example request (curl):** ```bash # For a CSV file curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_header/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/guess_header/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local" # For an Excel file curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/guess_header/?file_path=/tmp/username_abc123_financial_report.xlsx&file_type=excel&import_type=local&sheet_name=Q1_Results" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/guess_header/?file_path=/tmp/username_abc123_financial_report.xlsx&file_type=excel&import_type=local&sheet_name=Q1_Results" ``` -**Example using JavaScript:** +**Example request (JavaScript):** ```javascript const params = new URLSearchParams({ @@ -211,10 +212,10 @@ const params = new URLSearchParams({ import_type: 'local' }); -fetch(`https://demo.gethue.com/api/v1/importer/file/guess_header/?${params.toString()}`, { +fetch(`https://hue.example.com/api/v1/importer/file/guess_header/?${params.toString()}`, { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }) .then(response => response.json()) @@ -225,7 +226,7 @@ fetch(`https://demo.gethue.com/api/v1/importer/file/guess_header/?${params.toStr .catch(error => console.error('Error:', error)); ``` -**Response:** +**Example response:** ```json { @@ -233,63 +234,64 @@ fetch(`https://demo.gethue.com/api/v1/importer/file/guess_header/?${params.toStr } ``` -**How Header Detection Works:** +**How header detection works:** -The header detection algorithm uses multiple factors to make an educated guess: +Hue reads a sample of the file and uses the heuristic of the Python [`csv.Sniffer.has_header`](https://docs.python.org/3/library/csv.html#csv.Sniffer.has_header) method. For Excel files, it first converts the first 20 rows of the sheet to CSV. The result is a guess. Let the user confirm it. -1. Checks if the first row contains text while subsequent rows contain numeric values -2. Examines if the first row's data types differ from the majority of other rows -3. Looks for common header naming patterns (e.g., id, name, date, etc.) -4. Checks if the first row has no duplicate values while data rows might have duplicates - -**Status Codes:** +**Status codes:** - `200 OK` - Header detection successful - `400 Bad Request` - Invalid parameters or file not found - `500 Internal Server Error` - Server-side error -## Preview File - -Generate a preview of a file's content with column type mapping for creating SQL tables. +## Preview file -**Endpoint:** `/api/v1/importer/file/preview/` +Generate a preview of a file's content with column type mapping for creating SQL tables. The preview contains at most 50 rows. `preview_data` maps each column name to its list of values, and it is empty for an empty file. -**Method:** `GET` +**Method and URL:** `GET /api/v1/importer/file/preview/` -**Request Parameters:** +**Request parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| -| file_path | String | Yes | Full path to the file to preview | -| file_type | String | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | -| import_type | String | Yes | Type of import (`local` or `remote`) | -| sql_dialect | String | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | -| has_header | Boolean | Yes | Whether the file has a header row | -| sheet_name | String | No | Sheet name (required for Excel files) | -| field_separator | String | No | Field separator character (defaults to `,` for CSV, `\t` for TSV, required for `delimiter_format`) | -| quote_char | String | No | Quote character (defaults to `"`) | -| record_separator | String | No | Record separator character (defaults to `\n`) | - -**Example using cURL:** +| file_path | string | Yes | Full path to the file to preview | +| file_type | string | Yes | Type of file (`csv`, `tsv`, `excel`, `delimiter_format`) | +| import_type | string | Yes | Type of import (`local` or `remote`) | +| sql_dialect | string | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | +| has_header | boolean | Yes | Whether the file has a header row | +| sheet_name | string | No | Sheet name (required for Excel files) | +| field_separator | string | No | Field separator character (defaults to `,` for CSV, `\t` for TSV, required for `delimiter_format`) | +| quote_char | string | No | Quote character (defaults to `"`) | +| record_separator | string | No | Record separator character (defaults to `\n`). `\r\n` is treated as `\n`. | + +The `field_separator`, `quote_char` and `record_separator` values are read as escape sequences. Send the two characters `\t` for a tab and `\n` for a line feed. + +**Example request (curl):** ```bash # For a CSV file with header curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local&sql_dialect=hive&has_header=true" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_sales_data.csv&file_type=csv&import_type=local&sql_dialect=hive&has_header=true" # For an Excel file with header curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_financial_report.xlsx&file_type=excel&import_type=local&sql_dialect=impala&has_header=true&sheet_name=Q1_Results" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_financial_report.xlsx&file_type=excel&import_type=local&sql_dialect=impala&has_header=true&sheet_name=Q1_Results" # For a custom pipe-delimited file using delimiter_format -curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/file/preview/?file_path=/tmp/username_abc123_pipe_data.txt&file_type=delimiter_format&import_type=local&sql_dialect=hive&has_header=true&field_separator=|"e_char=\"&record_separator=\n" +curl -G "https://hue.example.com/api/v1/importer/file/preview/" \ + -H "Authorization: Bearer " \ + --data-urlencode "file_path=/tmp/username_abc123_pipe_data.txt" \ + --data-urlencode "file_type=delimiter_format" \ + --data-urlencode "import_type=local" \ + --data-urlencode "sql_dialect=hive" \ + --data-urlencode "has_header=true" \ + --data-urlencode "field_separator=|" \ + --data-urlencode 'record_separator=\n' ``` -**Example using JavaScript:** +**Example request (JavaScript):** ```javascript // Parameters for previewing a CSV file @@ -302,10 +304,10 @@ const params = new URLSearchParams({ }); // Make the fetch request -fetch(`https://demo.gethue.com/api/v1/importer/file/preview/?${params.toString()}`, { +fetch(`https://hue.example.com/api/v1/importer/file/preview/?${params.toString()}`, { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }) .then(response => response.json()) @@ -316,7 +318,7 @@ fetch(`https://demo.gethue.com/api/v1/importer/file/preview/?${params.toString() .catch(error => console.error('Error:', error)); ``` -**Response:** +**Example response:** For a sales data CSV file: ```json @@ -348,12 +350,14 @@ For a sales data CSV file: "type": "DATE" } ], - "preview_data": [ - ["1001", "Laptop XPS 13", "Electronics", "1299.99", "1", "2025-01-15"], - ["1002", "Wireless Headphones", "Electronics", "149.99", "2", "2025-01-15"], - ["1003", "Office Chair", "Furniture", "249.50", "1", "2025-01-16"], - ["1004", "Notebook Set", "Office Supplies", "24.95", "5", "2025-01-17"] - ] + "preview_data": { + "transaction_id": [1001, 1002, 1003, 1004], + "product_name": ["Laptop XPS 13", "Wireless Headphones", "Office Chair", "Notebook Set"], + "category": ["Electronics", "Electronics", "Furniture", "Office Supplies"], + "price": [1299.99, 149.99, 249.5, 24.95], + "quantity": [1, 2, 1, 5], + "purchase_date": ["2025-01-15", "2025-01-15", "2025-01-16", "2025-01-17"] + } } ``` @@ -394,41 +398,39 @@ The `delimiter_format` file type allows you to process custom delimited files th - Use `record_separator` from guess_metadata response - When `import_type` is set to `remote`, the API uses the file system associated with the user's request -**Status Codes:** +**Status codes:** - `200 OK` - Preview generated successfully - `400 Bad Request` - Invalid parameters or file not found - `500 Internal Server Error` - Server-side error -## Get SQL Type Mapping +## Get SQL type mapping Get the list of unique SQL data types supported by a specific SQL dialect. This helps in understanding what SQL types are available when creating tables in different SQL engines. -**Endpoint:** `/api/v1/importer/sql_type_mapping/` - -**Method:** `GET` +**Method and URL:** `GET /api/v1/importer/sql_type_mapping/` -**Request Parameters:** +**Request parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| -| sql_dialect | String | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | +| sql_dialect | string | Yes | SQL dialect for type mapping (`hive`, `impala`, `trino`, `phoenix`, `sparksql`) | -**Example using cURL:** +**Example request (curl):** ```bash curl -X GET \ - -H "Authorization: Bearer " \ - "https://demo.gethue.com/api/v1/importer/sql_type_mapping/?sql_dialect=hive" + -H "Authorization: Bearer " \ + "https://hue.example.com/api/v1/importer/sql_type_mapping/?sql_dialect=hive" ``` -**Example using JavaScript:** +**Example request (JavaScript):** ```javascript -fetch('https://demo.gethue.com/api/v1/importer/sql_type_mapping/?sql_dialect=hive', { +fetch('https://hue.example.com/api/v1/importer/sql_type_mapping/?sql_dialect=hive', { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }) .then(response => response.json()) @@ -439,7 +441,7 @@ fetch('https://demo.gethue.com/api/v1/importer/sql_type_mapping/?sql_dialect=hiv .catch(error => console.error('Error:', error)); ``` -**Response:** +**Example response:** The response is a sorted list of unique SQL data types supported by the specified dialect. @@ -557,57 +559,48 @@ Different SQL dialects support different sets of SQL types. Here are the unique ] ``` -**Key Differences Between Dialects:** +**Key differences between dialects:** - **Impala**: Does not support `INTERVAL` types (uses `STRING` for duration data) - **Trino**: Uses `INTEGER` instead of `INT`, `REAL` instead of `FLOAT`, `VARCHAR` instead of `STRING`, `ROW` instead of `STRUCT`, and supports `JSON` type - **Phoenix**: Supports unsigned integer types (`UNSIGNED_INT`, `UNSIGNED_LONG`, etc.) and has a dedicated `TIME` type -- **Spark SQL**: Similar to Hive but follows more standard SQL conventions +- **Spark SQL**: Uses the same types as Hive. -**Status Codes:** +**Status codes:** - `200 OK` - Mapping retrieved successfully - `400 Bad Request` - Invalid SQL dialect - `500 Internal Server Error` - Server-side error -## Error Handling +## Error handling -All endpoints follow a consistent error response format: +The API returns these status codes: -```json -{ - "error": "Error message describing what went wrong" -} -``` +- `400 Bad Request`: a parameter is missing or invalid, or the file does not exist. +- `401 Unauthorized`: the credentials are missing or invalid. +- `500 Internal Server Error`: the server failed to process the file. -**Example Error Responses:** +Parameter validation errors list the invalid fields: -File not found: ```json { - "error": "File /tmp/nonexistent_file.csv not found" + "sql_dialect": ["\"oracle\" is not a valid choice."] } ``` -Invalid parameter: -```json -{ - "error": "Sheet name is required for Excel files" -} -``` +Other errors return a message: -File size limit exceeded: ```json { - "error": "File too large. Maximum file size is 100 MiB." + "error": "Remote file does not exist: /user/hue/data/missing.csv" } ``` -Authentication errors will return a standard HTTP 401 Unauthorized response. +If an uploaded file is larger than the limit, the upload fails with `400 Bad Request`. The default limit is 150 MiB. An administrator can change it with `max_local_file_size_upload_limit` in the `[desktop] [[importer]]` section of `hue.ini`. -> **Note**: Authentication tokens can be obtained through Hue's authentication API. See the [Authentication API](/developer/api/rest/#authentication) documentation for details. +If a `guess_metadata` or `preview` call with `import_type=local` fails, Hue deletes the uploaded temporary file. Upload the file again before you retry. -## Complete Workflow Example +## Complete workflow example Here's an example workflow that combines all the APIs to import a CSV file into a Hive table: @@ -618,10 +611,10 @@ Here's an example workflow that combines all the APIs to import a CSV file into const formData = new FormData(); formData.append('file', fileInputElement.files[0]); -const uploadResponse = await fetch('https://demo.gethue.com/api/v1/importer/upload/file/', { +const uploadResponse = await fetch('https://hue.example.com/api/v1/importer/upload/file/', { method: 'POST', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' }, body: formData }); @@ -639,10 +632,10 @@ const metadataParams = new URLSearchParams({ import_type: 'local' }); -const metadataResponse = await fetch(`https://demo.gethue.com/api/v1/importer/file/guess_metadata/?${metadataParams.toString()}`, { +const metadataResponse = await fetch(`https://hue.example.com/api/v1/importer/file/guess_metadata/?${metadataParams.toString()}`, { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }); @@ -663,10 +656,10 @@ const headerParams = new URLSearchParams({ import_type: 'local' }); -const headerResponse = await fetch(`https://demo.gethue.com/api/v1/importer/file/guess_header/?${headerParams.toString()}`, { +const headerResponse = await fetch(`https://hue.example.com/api/v1/importer/file/guess_header/?${headerParams.toString()}`, { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }); @@ -689,10 +682,10 @@ const previewParams = new URLSearchParams({ record_separator: recordSeparator }); -const previewResponse = await fetch(`https://demo.gethue.com/api/v1/importer/file/preview/?${previewParams.toString()}`, { +const previewResponse = await fetch(`https://hue.example.com/api/v1/importer/file/preview/?${previewParams.toString()}`, { method: 'GET', headers: { - 'Authorization': 'Bearer ' + 'Authorization': 'Bearer ' } }); @@ -716,18 +709,9 @@ console.log(createTableSQL); // You can then execute this SQL using the SQL execution API ``` -## Best Practices - -1. **Error Handling**: Always implement proper error handling for API responses, especially for file uploads which may fail due to size limitations or network issues. - -2. **Progress Feedback**: For large file uploads, consider implementing progress indicators using XHR instead of fetch to track upload progress. - -3. **Type Checking**: When parsing and validating file data, always check that the inferred types match what you expect for your business data. - -4. **Sequential Processing**: Process API calls in the proper sequence as shown in the workflow example to ensure each step has the required data. - -5. **User Confirmation**: Allow users to review and modify the automatic type detection before creating tables, especially for large datasets. - -6. **Access Control**: Ensure file permissions are properly set for any uploaded files, especially in multi-user environments. +## Best practices -7. **Cleanup**: Consider implementing cleanup mechanisms for temporary uploaded files that are no longer needed. Files uploaded via the `/api/v1/importer/upload/file/` endpoint are stored in temporary locations (typically `/tmp/`) and should be cleaned up after processing. +1. Handle errors for every call. Uploads can fail on size or file type limits. +2. Call the endpoints in the order of the workflow. Each step needs the output of the previous one. +3. Let users review and change the detected header, separators and column types before you create a table. +4. Delete the temporary files you no longer need. Hue stores uploaded files in the server temporary directory. diff --git a/docs/docs-site/content/developer/components/_index.md b/docs/docs-site/content/developer/components/_index.md index f961c717afb..0f3b6b4fa90 100644 --- a/docs/docs-site/content/developer/components/_index.md +++ b/docs/docs-site/content/developer/components/_index.md @@ -14,20 +14,24 @@ There are two ways to get them: ### NPM registry -Published as a NPM package in https://www.npmjs.com/package/gethue. +Published as an NPM package: [gethue](https://www.npmjs.com/package/gethue). -The recommended way. Versioned, published on the stabdard public JavaSCript module registry NPM, without any dependency on a local Hue project. +The recommended way. Versioned, published on the standard public JavaScript module registry NPM, without any dependency on a local Hue project. To run the [demo app](https://github.com/cloudera/hue/tree/master/tools/examples/api/hue_dep) make the following changes: -1. Install gethue NPM package +1. Install the `gethue` NPM package: - npm install --save gethue + ```bash + npm install --save gethue + ``` 2. In [package.json](https://github.com/cloudera/hue/blob/master/tools/examples/api/hue_dep/package.json), remove `"hue": "file:../../../.."` without touching the newly added gethue dependency 3. In [src/app.js](https://github.com/cloudera/hue/blob/master/tools/examples/api/hue_dep/src/app.js), change the import line to: - import sqlAutocompleteParser from 'gethue/lib/parsers/hiveAutocompleteParser'; + ```js + import sqlAutocompleteParser from 'gethue/lib/parsers/hiveAutocompleteParser'; + ``` 4. In [webpack.config.js](https://github.com/cloudera/hue/blob/master/tools/examples/api/hue_dep/webpack.config.js): - Change `'js'` to `'node_modules/gethue'` under `resolve.modules` @@ -35,20 +39,26 @@ To run the [demo app](https://github.com/cloudera/hue/tree/master/tools/examples ### Local repository -Checkout Hue and cd to the [demo app](https://github.com/cloudera/hue/tree/master/tools/examples/api/hue_dep). Steps are similar if you would like to use your own app instead of the demo app. +Check out Hue and go to the [demo app](https://github.com/cloudera/hue/tree/master/tools/examples/api/hue_dep). The steps are similar if you would like to use your own app instead of the demo app. - cd tools/examples/api/hue_dep +```bash +cd tools/examples/api/hue_dep - npm install - npm run webpack - npm run app +npm install +npm run webpack +npm run app +``` -In hue_dep `package.json` there is a dependency on Hue: +In the `hue_dep` `package.json` there is a dependency on Hue: - "dependencies": { - "hue": "file:../../.." - }, +```json +"dependencies": { + "hue": "file:../../../.." +}, +``` -Now let's import the Hive parser: +Now import the Hive parser: - import sqlAutocompleteParser from 'hue/desktop/core/src/desktop/js/parse/sql/hive/hiveAutocompleteParser'; +```js +import sqlAutocompleteParser from 'hue/desktop/core/src/desktop/js/parse/sql/hive/hiveAutocompleteParser'; +``` diff --git a/docs/docs-site/content/developer/components/parsers/_index.md b/docs/docs-site/content/developer/components/parsers/_index.md index 4ef7ab20b5c..26e0045e988 100644 --- a/docs/docs-site/content/developer/components/parsers/_index.md +++ b/docs/docs-site/content/developer/components/parsers/_index.md @@ -6,7 +6,7 @@ weight: 2 The parsers are the flagship part of Hue and power advanced autocompletes and other [SQL functionalities](/user/querying/#autocomplete). -Refer the [SQL Parser Documentation](/developer/development/#sql-parsers) for how they works and are built. +Refer to the [SQL parsers documentation](/developer/development/#sql-parsers) to learn how they work and how they are built. ## Live Demo @@ -63,22 +63,19 @@ Refer the [SQL Parser Documentation](/developer/development/#sql-parsers) for ho ## Import -Import the parser you need with something like below and run it on an SQL statement: +Install the [gethue](https://www.npmjs.com/package/gethue) NPM package (`npm install --save gethue`). Then import the parser you need with something like below and run it on an SQL statement: - import sqlAutocompleteParser from 'gethue/parse/sql/hive/hiveAutocompleteParser'; +```js +import sqlAutocompleteParser from 'gethue/lib/parsers/hiveAutocompleteParser'; - const beforeCursor = 'SELECT col1, col2, tbl2.col3 FROM tbl; '; // Note extra space at end - const afterCursor = ''; - const dialect = 'hive'; - const debug = false; +const beforeCursor = 'SELECT col1, col2, tbl2.col3 FROM tbl; '; // Note extra space at end +const afterCursor = ''; +const debug = false; - console.log( - JSON.stringify( - sqlAutocompleteParser.parseSql(beforeCursor, afterCursor, dialect, debug), - null, - 2 - ) - ); +console.log( + JSON.stringify(sqlAutocompleteParser.parseSql(beforeCursor, afterCursor, debug), null, 2) +); +``` Which then will output keywords suggestions and all the known locations: diff --git a/docs/docs-site/content/developer/components/scratchpad/_index.md b/docs/docs-site/content/developer/components/scratchpad/_index.md index da8f078607c..d90871e9d9e 100644 --- a/docs/docs-site/content/developer/components/scratchpad/_index.md +++ b/docs/docs-site/content/developer/components/scratchpad/_index.md @@ -27,33 +27,37 @@ The Scratchpad is hosted within this page and points to [demo.gethue.com](https: ## Importing -First install the NPM [package](/developer/components/) and import the component lib and its HTML element directly into the Web page: +First install the [NPM package](/developer/components/#npm-registry) and import the component lib and its HTML element directly into the Web page: - - +```html + + - - SQL Scratchpad - - + + SQL Scratchpad + + - -
- -
- + +
+ +
+ - + +``` -Then make sure *api-url* points to a running Hue or [Compose](https://github.com/gethue/compose) APIs. The API is the middleware between your Data Warehouse and Web Browser client and will provide the dynamic content like the list of tables and columns and enrich the autocomplete. The autocomplete is static and powered by the [parser](/developer/components/parsers/) selected by the value of *dialect*. +Then make sure `api-url` points to a running Hue server. The API is the middleware between your Data Warehouse and Web Browser client and will provide the dynamic content like the list of tables and columns and enrich the autocomplete. The autocomplete is static and powered by the [parser](/developer/components/parsers/) selected by the value of `dialect`. -Specify the credentials for the authentication via *username="demo" password="demo"*. +Specify the credentials for the authentication via `username` and `password` (e.g. `username="demo" password="demo"`). -Options to look-up a local JWT token or go via Login modal and so not requiring the credentials are coming shortly. +Options to look up a local JWT token, or to use a login modal and so not require the credentials, are coming shortly. -It is also possible to skip the whole `npm install` and directly grab the module via: +You can also skip `npm install` and load the module directly via: - +```html + +``` !["SQL Scratchpad"](https://cdn.gethue.com/uploads/2021/05/sql-scratchpad-v0.5.png) diff --git a/docs/docs-site/content/developer/development/_index.md b/docs/docs-site/content/developer/development/_index.md index 829a5b39d7f..710f39072da 100644 --- a/docs/docs-site/content/developer/development/_index.md +++ b/docs/docs-site/content/developer/development/_index.md @@ -5,128 +5,156 @@ draft: false weight: 1 --- -This section goes into greater detail on how to build and reuse the components of Hue. +This section goes into greater detail on how to build, test, extend and release Hue. +## Quick start -## Quick Start +### Requirements -### Dependencies +* The OS specific packages listed in the [install guide](/administrator/installation/dependencies/) +* Python 3.11 (the only supported Python version) +* Node.js 22.13 or later, with npm (see the `engines` field of `package.json`) -* The OS specific packages are listed in the [install guide](/administrator/installation/dependencies/) -* Python 3.11 -* Vue.js 3 -* Node.js ([22.13+](https://deb.nodesource.com/setup_22.x)) +### Build & start -### Build & Start +Build once, from the root of your Hue clone: -Build once: +```bash +export PYTHON_VER=python3.11 - export PYTHON_VER=python3.11 +# On macOS you might need to set +export SKIP_PYTHONDEV_CHECK=true - # Mac user might need to set - export SKIP_PYTHONDEV_CHECK=true +make apps +``` - # Export ROOT which should point to your Hue directory - export ROOT= +The [dependencies documentation](/administrator/installation/dependencies/) helps with troubleshooting build issues. - make apps +Create your local configuration file from the template: -The [dependencies documentation](/administrator/installation/dependencies/) is here to help for troubleshooting build issues. +```bash +cp desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini +``` -Then start the dev server (which will auto reload on file changes): +Then start the dev server, which reloads automatically when Python files change: - ./build/env/bin/hue runserver +```bash +./build/env/bin/hue runserver +``` -If you are changing JavaScript or CSS files, also start: +If you are changing frontend files (TypeScript, JavaScript, Vue, SCSS), also start the webpack watcher in another terminal: - npm run dev +```bash +npm run dev +``` -Persistence: Hue needs an existing database with transactions like MySQL to support concurrent requests and also not lose the registered users, saved queries, sharing permissions… when the server gets stopped. +### Use a transactional database -Once build, in order to avoid the `database is locked` errors, you also need to connect Hue to a transactional database. It is recommended to use MySQL or PostGreSQL for development. +Hue needs a transactional database like MySQL or PostgreSQL to support concurrent requests and to keep the registered users, saved queries and sharing permissions when the server stops. With the default SQLite database you might see `database is locked` errors. -Open the `desktop/conf/pseudo-distributed.ini` file in a text editor. Add the following options (and modify accordingly) for your MySQL setup: +Open the `desktop/conf/pseudo-distributed.ini` file in a text editor. Directly below the `[[database]]` line, add the following options (and modify them for your MySQL setup): -Directly below the `[[database]]` line, add the following. +```ini +host=localhost +port=3306 +engine=mysql +user=hue +password=secretpassword +name=hue +``` - host=localhost - port=3306 - engine=mysql - user=hue - password=secretpassword - name=hue Read more about how [configurations work](/administrator/configuration/). -### First SQL queries! +### First SQL queries -Here is how to point the Editor to either MySql or [Apache Hive](https://hive.apache.org/) and execute your first SQL queries. For other supported databases refer to the [connectors](/administrator/configuration/connectors/). +Here is how to point the Editor to either MySQL or [Apache Hive](https://hive.apache.org/) and execute your first SQL queries. For other supported databases refer to the [connectors](/administrator/configuration/connectors/). #### MySQL -In `desktop/conf/pseudo-distributed.ini`, below the `[[interpreters]]` section of of `[notebook]`. +In `desktop/conf/pseudo-distributed.ini`, below the `[[interpreters]]` section of `[notebook]`, add: - [[[mysql]]] - name=MySQL - interface=sqlalchemy - options='{"url": "mysql://${USER}:${PASSWORD}@localhost:3306/hue"}' +```ini +[[[mysql]]] +name=MySQL +interface=sqlalchemy +options='{"url": "mysql://${USER}:${PASSWORD}@localhost:3306/hue"}' +``` #### Apache Hive -You can connect to an existing Hive instance or setup a new one locally. An easy way to setup one is with Docker. You could use this [Hive 2 container](https://github.com/big-data-europe/docker-hive.). +You can connect to an existing Hive instance or setup a new one locally. An easy way to setup one is with Docker. You could use this [Hive 2 container](https://github.com/big-data-europe/docker-hive). - git clone https://github.com/big-data-europe/docker-hive - cd docker-hive - docker-compose up -d +```bash +git clone https://github.com/big-data-europe/docker-hive +cd docker-hive +docker-compose up -d +``` -Just follow the above 3 steps and you would have a running Hive instance on `jdbc:hive2://localhost:10000`. +Those three commands give you a running Hive instance on `jdbc:hive2://localhost:10000`. Now under Hue open `desktop/conf/pseudo-distributed.ini` file in a text editor, and modify the following properties: 1. Directly below the `[[beeswax]]` line, add the following: - # Host where HiveServer2 is running. - hive_server_host=localhost - # Port where HiveServer2 Thrift server runs on. - hive_server_port=10000 - thrift_version=7 + ```ini + # Host where HiveServer2 is running. + hive_server_host=localhost + # Port where HiveServer2 Thrift server runs on. + hive_server_port=10000 + thrift_version=7 + ``` 2. Below the `[[interpreters]]` of `[notebook]`, add: - [[[hive]]] - name=Hive - interface=hiveserver2 + ```ini + [[[hive]]] + name=Hive + interface=hiveserver2 + ``` + And restart Hue, open the Editors and start typing your first queries! -### Dev environment +### Development environment #### Lint configs +* [pyproject.toml](https://github.com/cloudera/hue/blob/master/pyproject.toml): Ruff (Python lint and format), pytest and coverage settings * [.eslintrc.js](https://github.com/cloudera/hue/blob/master/.eslintrc.js) -* [.pylintrc](https://github.com/cloudera/hue/blob/master/.pylintrc) * [.prettierrc](https://github.com/cloudera/hue/blob/master/.prettierrc) -* [Git hooks](https://github.com/cloudera/hue/blob/master/tools/githooks) +* [.stylelintrc](https://github.com/cloudera/hue/blob/master/.stylelintrc) +* Git hooks: [commit message hook](https://github.com/cloudera/hue/blob/master/tools/githooks/commit-msg) and [pre-commit / pre-push hooks](https://github.com/cloudera/hue/tree/master/tools/git-hooks) -For checking git commit message format automatically locally: +To check the commit message format locally, install the commit message hook: - cp tools/githooks/* .git/hooks - chmod +x .git/hooks/* +```bash +cp tools/githooks/commit-msg .git/hooks/commit-msg +chmod +x .git/hooks/commit-msg +``` + +To also run ESLint on the staged JavaScript and TypeScript files before each commit, install the pre-commit hook (it installs ESLint globally if missing): + +```bash +./setup-hooks.sh +``` -#### Visual Studio Code -Add the 'hue' directory as a workspace, then install the following extensions: +#### Visual Studio Code -Recommended extensions: +Add the `hue` directory as a workspace, then install the recommended extensions: * [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Microsoft -* [EsLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - Dirk Baeumur +* [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) - Astral Software +* [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - Microsoft +* [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) - Prettier * [Mako](https://marketplace.visualstudio.com/items?itemName=tommorris.mako) - tommorris * [Docker](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) - Microsoft #### PyCharm -First step is to configure pycharm to use the Hue virtual environment at ./build/env/env +First step is to configure pycharm to use the Hue virtual environment at `./build/env` + ![Pycharm virtualenv](/images/pycharm_virtualenv.png) Second step is to configure the debug configuration @@ -134,7 +162,7 @@ Second step is to configure the debug configuration #### Eclipse -First step is to configure Eclipse to use the Hue virtual environment at ./build/env/env +First step is to configure Eclipse to use the Hue virtual environment at `./build/env` ![Eclipse interpreter](/images/eclipse_interpreter.png) Second step is to configure the debug configuration @@ -143,104 +171,126 @@ Second step is to configure the debug configuration ![Eclipse debug interpreter](/images/eclipse_debug_interpreter.png) -## Development Process - -**Note:** +## Development process -During the development process if you are facing any problem then, it is recommended to search for information on the [Forum](https://discourse.gethue.com/) and in the [bug tracker](https://github.com/cloudera/hue/issues?q=is%3Aissue+). +If you face a problem, search the [Forum](https://discourse.gethue.com/) and the [bug tracker](https://github.com/cloudera/hue/issues?q=is%3Aissue+) first. -Here is a tutorial about how to sent a patch request for review. +This section explains how to send a change for review. ### Setup -Hue project uses GitHub Pull Requests (PR) for code reviews. It also automatically runs the CI (syntax check, tests...) for you. +The Hue project uses GitHub pull requests (PRs) for code reviews. The CI (build, lint, tests) runs automatically on each PR. -If you've never used git and github before, there are bunch of things you need to [do](https://kbroman.org/github_tutorial/pages/first_time.html) before going further. +If you have never used Git and GitHub, follow this [first-time setup guide](https://kbroman.org/github_tutorial/pages/first_time.html) before going further. -Now, clone cloudera/hue: +Clone [cloudera/hue](https://github.com/cloudera/hue) (or your fork): - git clone https://github.com/cloudera/hue +```bash +git clone https://github.com/cloudera/hue +``` -Create a new branch with the Github issue GH-XXX or explicit name as the branch name: +Create a branch named after the GitHub issue (`GH-XXX`) or with an explicit name: - git checkout master - git pull --rebase origin master - git checkout -b GH-XXX +```bash +git checkout master +git pull --rebase origin master +git checkout -b GH-XXX +``` -Then make your changes in code: +Make your changes, then commit them: - git add - git diff --cached - git commit -m "GH-XXX " +```bash +git add +git diff --cached +git commit -m "GH-XXX [hive] Add numExecutors option" +``` -### Post a review +### Commit message format -Either post via the [GitHub CLI](https://github.com/cli/cli): +The first line of each commit is checked by the CI with [`tools/ci/check_for_commit_message.sh`](https://github.com/cloudera/hue/blob/master/tools/ci/check_for_commit_message.sh), which relies on the [commit-msg hook](https://github.com/cloudera/hue/blob/master/tools/githooks/commit-msg). The only enforced rule is a maximum of 71 characters (50 recommended). By convention, start with a label for the area you are changing: - gh pr create --fill --assignee=romainr --web + [hive] Add numExecutors option + [hive] Add numExecutors option (#1000) + GH-1000 [hive] Add numExecutors option -Or push to your branch in your repository fork, e.g. if you are 'bob': +Common labels are `[hive]`, `[impala]`, `[ui]`, `[docs]`, `[api]`, `[core]`, `[trino]`. - git remote add bob https://github.com/bob/hue - git fetch bob +### Post a review - git push bob HEAD:GH-1000-fix +Either create the pull request with the [GitHub CLI](https://github.com/cli/cli): -And create the pull request to cloudera/hue via the button on the https://github.com/bob/hue/tree/GH-1000-fix page. +```bash +gh pr create --fill --web +``` -**Note**: -GitHub CLI is recommended for automating even more the process. e.g. for asking reviewers and auto merging when all the checks are green: +Or push your branch to your fork, for example if you are `bob`: - gh pr create --fill --assignee=romainr --reviewer=agl29,Harshg999 - gh pr merge --auto -r 2494 +```bash +git remote add bob https://github.com/bob/hue +git fetch bob +git push bob HEAD:GH-1000-fix +``` + +Then open the pull request to `cloudera/hue` from the `https://github.com/bob/hue/tree/GH-1000-fix` page. + +With the GitHub CLI, you can also ask for reviewers and enable auto-merge once all the checks are green: + +```bash +gh pr create --fill --reviewer= +gh pr merge --auto --rebase +``` ### Update a review -Modify the previous commit diff: +Modify the previous commit: - git add - git commit --amend +```bash +git add +git commit --amend +``` -Update the review: +Update the pull request: - git push bob HEAD:ISSUE-1000-fix -f +```bash +git push bob HEAD:GH-1000-fix -f +``` -And it will automatically kick the CI and notify reviewers. +This re-runs the CI and notifies the reviewers. -### Ship It +### Merge -Once we get ship it from at least two reviewer, we can push the changes to master +Once the pull request is approved and the CI checks are green, a maintainer merges it into `master`. Before that, keep your branch up to date: - git rebase origin/master - git push origin HEAD:ci-commit-master- +```bash +git fetch origin +git rebase origin/master +``` -* The push will auto run the tests and push it to master -* It can be seen on https://github.com/cloudera/hue/actions - * Python 3 CI should be green on Python 3.11 - * If successful, the change would be auto merged to master - * On failure, we will get a mail - * Runs usually take 10-20 min -* Once merged mark the review as submitted - **Close > Submitted** -* Add the commit link to the ticket and mark it as resolved +You can follow the CI runs on [GitHub Actions](https://github.com/cloudera/hue/actions). The workflows are in [`.github/workflows`](https://github.com/cloudera/hue/tree/master/.github/workflows): +| Workflow | What it checks | +| -------- | -------------- | +| Python 3 CI (`commitflow-py3.yml`) | Commit titles, build on Python 3.11, `pytest`, `hue runruff check` | +| Frontend CI (`commitflow-frontend.yml`) | Commit titles, pinned versions, licenses, `prettier`, `eslint`, `stylelint`, `jest` | +| Docs Lint (`docs_lint.yml`) | Dead links in `docs/docs-site` (runs on docs changes) | ### Sum-up -We hope that these commands will make your life easier and encourage you to [contribute to Hue](https://github.com/cloudera/hue/blob/master/CONTRIBUTING.md) 😉 +We hope that these commands make your life easier and encourage you to [contribute to Hue](https://github.com/cloudera/hue/blob/master/CONTRIBUTING.md). -As usual feel free to send feedback on the [Forum](https://discourse.gethue.com/) list or [GitHub issues](https://github.com/cloudera/hue/issues)! +As usual, feel free to send feedback on the [Forum](https://discourse.gethue.com/) list or [GitHub issues](https://github.com/cloudera/hue/issues)! -## API Server +## API server -### The Hue Server +### The Hue server ![Web Back-end](/images/webbackend.png) Hue is a web application built on the Django python web framework. -Django, running on the WSGI container/web server (typically CherryPy), manages +Django, running on a WSGI container/web server (Gunicorn or CherryPy), manages the url dispatch, executes application logic code, and puts together the views -from their templates. Django uses a database (typically MySql or PostGres) to manage session data, and Hue applications can use it as well +from their templates. Django uses a database (typically MySQL or PostgreSQL) to manage session data, and Hue applications can use it as well for their "models". (For example, the saved Editor stores saved queries in the database.) In addition to the web server, some Hue applications run @@ -248,244 +298,257 @@ daemon processes "on the side". Some examples are the `Celery Task Server`, `Cel ![Reference Architecture](/images/hue_architecture.png) -### File Layout - -The Hue "framework" is in ``desktop/core/`` and contains the Web components. -``desktop/libs/`` is the API for talking to various Hadoop services. -The installable apps live in ``apps/``. Please place third-party dependencies in the app's ext-py/ -directory. - -The typical directory structure for inside an application includes: -``` - src/ - for Python/Django code - models.py - urls.py - views.py - forms.py - settings.py +### File layout - conf/ - for configuration (``.ini``) files to be installed +The Hue "framework" is in `desktop/core/` and contains the Web components. +`desktop/libs/` is the API for talking to various Hadoop services. +The installable apps live in `apps/`. Third-party Python packages are listed in +`desktop/core/base_requirements.txt`. A few vendored packages live in `desktop/core/ext-py3/`. - static/ - for static HTML/js resources and help doc +The typical directory structure inside an application includes: - templates/ - for data to be put through a template engine - - locales/ - for localizations in multiple languages +```text +src/ Python/Django code (models.py, urls.py, views.py, forms.py, settings.py) +conf/ configuration (.ini) files to be installed +static/ static HTML/JS resources and help docs +templates/ data to be put through a template engine +locales/ localizations in multiple languages ``` -For the URLs within your application, you should make your own ``urls.py`` -which will be automatically rooted at ``/yourappname/`` in the global -namespace. See ``apps/about/src/about/urls.py`` for an example. +For the URLs within your application, make your own `urls.py` +which will be automatically rooted at `/yourappname/` in the global +namespace. See `apps/about/src/about/urls.py` for an example. ### Configurations #### File Hue uses a typed configuration system that reads configuration files (in an -ini-style format). By default, Hue loads all `*.ini` files in the `build/desktop/conf` -directory. The configuration files have the following format: - - # This is a comment - [ app_name ] # Same as your app's name - app_property = "Pink Floyd" - - [[ section_a ]] # The double brackets start a section under [ app_name ] - a_weight = 80 # that is useful for grouping - a_height = 180 +ini-style format). By default, Hue loads all `*.ini` files in the `desktop/conf` +directory (or the directory pointed to by the `HUE_CONF_DIR` environment variable). The configuration files have the following format: + +```ini +# This is a comment +[ app_name ] # Same as your app's name +app_property = "Pink Floyd" + +[[ section_a ]] # The double brackets start a section under [ app_name ] +a_weight = 80 # that is useful for grouping +a_height = 180 + +[[ filesystems ]] # Sections are also useful for making a list +[[[ cluster_1 ]]] # All list members are sub-sections of the same type +namenode_host = localhost +# User may define more: +# [[[ cluster_2 ]]] +# namenode_host = 10.0.0.1 +``` - [[ filesystems ]] # Sections are also useful for making a list - [[[ cluster_1 ]]] # All list members are sub-sections of the same type - namenode_host = localhost - # User may define more: - # [[[ cluster_2 ]]] - # namenode_host = 10.0.0.1 #### Variables Your application's `conf.py` is special. It provides access to the configuration file (and even default configurations not specified in the file). Using the above example, your `conf.py` should -define the following: +define the following. A `desktop.lib.conf.Config` object for `app_property`, such as: - MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah') +```python +MY_PROPERTY = Config(key='app_property', default='Beatles', help='blah') +``` - You can access its value by `MY_PROPERTY.get()`. +You can access its value with `MY_PROPERTY.get()`. A `desktop.lib.conf.ConfigSection` object for `section_a`, such as: - SECTION_A = ConfigSection(key='section_a', - help='blah', - members=dict( - AWEIGHT=Config(key='a_weight', type=int, default=0), - AHEIGHT=Config(key='a_height', type=int, default=0))) +```python +SECTION_A = ConfigSection( + key='section_a', + help='blah', + members=dict( + AWEIGHT=Config(key='a_weight', type=int, default=0), + AHEIGHT=Config(key='a_height', type=int, default=0))) +``` - You can access the values by `SECTION_A.AWEIGHT.get()`. +You can access the values with `SECTION_A.AWEIGHT.get()`. A `desktop.lib.conf.UnspecifiedConfigSection` object for `filesystems`, such as: - FS = UnspecifiedConfigSection( - key='filesystems', - each=ConfigSection(members=dict( - nn_host=Config(key='namenode_host', required=True)) +```python +FS = UnspecifiedConfigSection( + key='filesystems', + each=ConfigSection(members=dict( + nn_host=Config(key='namenode_host', required=True)))) +``` + ### Saving documents Each app used to have its own model to store its data (e.g. a saving a SQL query, query history...). All the models have been unified into a single Document2 model in the desktop app: -``desktop/core/src/desktop/models.py``. +`desktop/core/src/desktop/models.py`. -The `Document2` model provides automatically creation, sharing and saving. It persists the document data into a json field, which limits the need ot database migrations and simplifies the interaction with the frontend. +The `Document2` model provides automatically creation, sharing and saving. It persists the document data into a json field, which limits the need for database migrations and simplifies the interaction with the frontend. -`Document2` is based on [Django Models](https://docs.djangoproject.com/en/1.11/#the-model-layer) -are Django's Object-Relational Mapping framework. +`Document2` is based on [Django models](https://docs.djangoproject.com/en/5.2/#the-model-layer), +Django's Object-Relational Mapping (ORM) framework. -### Authentication Backends +### Authentication backends Hue exposes a configuration flag ("auth") to configure a custom authentication [backend](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/auth/backend.py). -See [writing an authentication backend](http://docs.djangoproject.com/en/dev/topics/auth/#writing-an-authentication-backend) +See [writing an authentication backend](https://docs.djangoproject.com/en/5.2/topics/auth/customizing/#writing-an-authentication-backend) for more details. In addition to that, backends may support a `manages_passwords_externally()` method, returning True or False, to tell the user manager application whether or not changing passwords within Hue is possible. -### Using and Installing Thrift +### Using and installing Thrift Right now, we check in the generated thrift code. To generate the code, you'll need the thrift binary version 0.9.0. Please download from http://thrift.apache.org/. -The modules using ``Thrift`` have some helper scripts like ``regenerate_thrift.sh`` +The modules using Thrift have some helper scripts like `regenerate_thrift.sh` for regenerating the code from the interfaces. ### Upgrades -After upgrading the version of Hue, running these two commands will make sure the database has the correct tables and fields. +After upgrading the version of Hue, run this command to make sure the database has the correct tables and fields: - ./build/env/bin/hue migrate +```bash +./build/env/bin/hue migrate +``` -### Debugging Tips and Tricks +### Debugging tips and tricks * Set `DESKTOP_DEBUG=1` as an environment variable if you want logs to go to stderr as well as to the respective log files. -* Use runserver. If you want to set a CLI breakpoint, just insert - `__import__("ipdb").set_trace()` - into your code. +* Use `runserver`. If you want to set a CLI breakpoint, insert + `breakpoint()` into your code. * Django tends to restart its server whenever it notices a file changes. For certain things (like configuration changes), this is not sufficient. Restart the server whole-heartedly. -* We recommend developing with the Chrome console. +* We recommend developing with the browser developer tools open. -* Special environment variables +Special environment variables: - DESKTOP_LOGLEVEL= - level can be DEBUG, INFO, WARN, ERROR, or CRITICAL. - When specified, the console logger is set to the given log level. A console - logger is created if one is not defined. +```text +DESKTOP_LOGLEVEL= + level can be DEBUG, INFO, WARN, ERROR, or CRITICAL. + When specified, the console logger is set to the given log level. A console + logger is created if one is not defined. - DESKTOP_DEBUG - A shorthand for DESKTOP_LOG_LEVEL=DEBUG. Also turns on output HTML - validation. +DESKTOP_DEBUG + A shorthand for DESKTOP_LOGLEVEL=DEBUG. - DESKTOP_PROFILE - Turn on Python profiling. The profile data is saved in a file. See the - console output for the location of the file. +DESKTOP_PROFILE + Turn on Python profiling. The profile data is saved in a file. See the + console output for the location of the file. - DESKTOP_LOG_DIR=$dir - Specify the HUE log directory. Defaults to ./log. +DESKTOP_LOG_DIR=$dir + Specify the Hue log directory. - DESKTOP_DB_CONFIG=$db engine:db name:test db name:username:password:host:port - Specify alternate DB connection parameters for HUE to use. Useful for - testing your changes against, for example, MySQL instead of sqlite. String - is a colon-delimited list. +DESKTOP_DB_CONFIG=$db engine:db name:test db name:username:password:host:port + Specify alternate DB connection parameters for Hue to use. Useful for + testing your changes against, for example, MySQL instead of SQLite. The + string is a colon-delimited list. +``` - TEST_IMPALAD_HOST=impalad-01.gethue.com - Point to an Impalad and trigger the Impala tests. -## User Interface (UI) +## User interface (UI) Developing applications for Hue requires a minimal amount of CSS (and potentially JavaScript) to use existing functionality. -In a nutshell, front-end development is using: +In a nutshell, frontend development uses: -* [Vue.js](https://vuejs.org/) to script the custom interactions -* TypeScript -* [Bootstrap](https://getbootstrap.com/) to layout your app -* [Mako](http://www.makotemplates.org/) is the templating language (currently being removed in favor of Vue.js)) +* [React](https://react.dev/) with [TypeScript](https://www.typescriptlang.org/) for all new UI, built with [Ant Design](https://ant.design/) and Cloudera UI (`cuix`) components. See the [frontend contributing guide](https://github.com/cloudera/hue/blob/master/contributing-frontend.md). +* SCSS (using BEM naming) for new styles +* [Mako](https://www.makotemplates.org/) as the server-side templating language +* Legacy code in Knockout.js, jQuery, Vue.js and [Bootstrap](https://getbootstrap.com/) 2, which is progressively being migrated to React -### Javascript +### JavaScript and TypeScript -The javascript files are currently being migrated to webpack bundles, during this process some files will live under src/desktop/static/ and some will live under src/desktop/js +The frontend source lives in `desktop/core/src/desktop/js` and is built into webpack bundles. Third-party libraries that are not managed by npm live under `desktop/core/src/desktop/static/desktop/ext`. -For changes to the files under src/desktop/js the following applies: +First make sure all third-party dependencies defined in `package.json` are installed into `node_modules/`: -First make sure all third-party dependencies defined in package.json are installed into node_modules/ +```bash +npm install +``` - npm install +Run this again after changing `package.json`. -Also run this after making changes to package.json, adding new third-party dependencies etc. +To generate the bundles run: -To generate the js bundles run: +```bash +npm run webpack +npm run webpack-workers +npm run webpack-login +``` - npm run webpack - npm run webpack-workers - npm run webpack-login +During development, the bundles are regenerated when files change with: -During development the bundles can be autogenerated when it detects changes to the .js files, for this run: +```bash +npm run dev +``` - npm run dev +If you work on the web workers or the login page, also run `npm run dev-workers` or `npm run dev-login`. -Before sending a review with changes to the bundles run: +Before sending a review, run the checks that the CI runs and fix the issues they report: - npm run lint-fix +```bash +npm run lint-fix +npm run prettier-fix +npm run style-lint-fix +``` -and possibly fix any issues it might report. +### CSS and SCSS -### CSS / LESS +Write new styles in SCSS (`.scss` files next to the component, with BEM class names) and use the existing variables instead of hardcoded colors. Do not add new Less files; migrate a Less file to SCSS when you touch it. -Hue uses [Bootstrap](https://getbootstrap.com/) version 2.0 CSS -styles and layouts. They are highly reusable and flexible. Your app doesn't -have to use these styles, but if you do, it'll save you some time and make your -app look at home in Hue. +SCSS files under `desktop/core/src/desktop/js` are bundled by webpack. Lint them with: -After changing the CSS in a .less file, rebuilding with: +```bash +npm run style-lint +``` - npm run less +Legacy Less files live in `desktop/core/src/desktop/static/desktop/less` and are compiled with Grunt. Hue also uses [Bootstrap](https://getbootstrap.com/) 2 CSS styles and layouts in the legacy pages. After changing a `.less` file, rebuild the CSS with: -Or run in watch mode that will generate the .css on any change to the .less files: +```bash +npm run less +``` - npm run less-dev +Or run in watch mode to regenerate the `.css` on any change to the `.less` files: -After less changes make sure linting is run with: +```bash +npm run less-dev +``` - npm run less-lint ### Icons You should create an icon for your application that is a transparent png sized 24px by 24px. Your `settings.py` file should point to your icon via the `ICON` -variable. The `create_desktop_app` command creates a default icon for you. +variable. The `create_desktop_app` command sets `ICON` in the `settings.py` of the new app. Put your icon at that path. **NOTE:** If you do not define an application icon, your application will not show up in the navigation bar. -Hue ships with Twitter Bootstrap and Font Awesome 4 so you have plenty of scalable icons to choose from. You can style your elements to use them like this (in your mako template): +Hue ships with Twitter Bootstrap and Font Awesome 4 so you have plenty of scalable icons to choose from. You can style your elements to use them like this (in your Mako template): - - Trash +```html + + Trash +``` ### Static files -For better performances, Hue uses the Django staticfiles app. If in production mode, if you edit -some static files, you would need to run this command or `make apps`. No actions are needed in +For better performance, Hue uses the Django staticfiles app. In production mode, if you edit +some static files, you need to run this command or `make apps`. No action is needed in development mode. -``` + +```bash ./build/env/bin/hue collectstatic ``` @@ -493,104 +556,126 @@ development mode. ### The short story -Run the API unit tests - - ./build/env/bin/pytest +Prerequisite: build Hue and install the test dependencies once: -When opening a pull request, it will automatically trigger a [Github Action](https://github.com/cloudera/hue/actions/) that includes running the unit tests. - -Run the user interface tests: - - npm run test - -### Running the API tests +```bash +export PYTHON_VER=python3.11 +make apps test_prep +``` -In the pyproject.toml file, we define the configuration for pytest, including settings to enable pytest to discover and execute the unit tests. +Run the API unit tests: -For pytest to recognize test files, you should follow the naming convention `*_test.py` or `test_*.py`. Following the pytest framework convention for test discovery is recommended. Pytest will gather test items according to the following rules: +```bash +./build/env/bin/pytest +``` -1. Test functions or methods prefixed with test_ outside of a class. -2. Test functions or methods prefixed with test_ inside classes prefixed with Test. +Run the user interface tests: -You can refer to `desktop/libs/notebook/src/notebook/connectors/trino_tests.py` for an example of how tests are organized. +```bash +npm run test +``` -To run the unit tests with pytest (should not take more than 5-10 minutes): +When you open a pull request, [GitHub Actions](https://github.com/cloudera/hue/actions/) automatically runs the unit tests, the linters and the commit message check. - ./build/env/bin/pytest +### Running the API tests -To run the tests of a particular app, use: +The pytest configuration (test discovery, markers, report and coverage options) is in the `[tool.pytest.ini_options]` section of `pyproject.toml`. By default, tests marked `integration` are skipped. - ./build/env/bin/pytest test_app_path +Pytest discovers the files named `tests.py`, `test_*.py`, `*_tests.py`, `tests_*` or `*_test.py`. It then collects: -e.g. +1. Test functions or methods prefixed with `test_` outside of a class. +2. Test functions or methods prefixed with `test_` inside classes prefixed with `Test`. - ./build/env/bin/pytest apps/filebrowser +Refer to `desktop/libs/notebook/src/notebook/connectors/trino_tests.py` for an example of how tests are organized. -To run the tests of a specific class, use: +Run all the unit tests (this can take several minutes): - ./build/env/bin/pytest test_file_path::class +```bash +./build/env/bin/pytest +``` -e.g. +Run the tests of a particular app or directory: - ./build/env/bin/pytest apps/useradmin/src/useradmin/tests.py::TestUserAdmin +```bash +./build/env/bin/pytest apps/filebrowser +``` -To run a specific test in a class, use: +Run the tests of a specific class: - ./build/env/bin/pytest test_file_path::class::function +```bash +./build/env/bin/pytest apps/useradmin/src/useradmin/tests.py::TestUserAdmin +``` -e.g. +Run a specific test in a class: - ./build/env/bin/pytest apps/useradmin/src/useradmin/tests.py::TestUserAdmin::test_user_admin +```bash +./build/env/bin/pytest apps/useradmin/src/useradmin/tests.py::TestUserAdmin::test_user_admin +``` -Note: +If a test fails with an error similar to: -When running the tests and seeing an error similar to: +```text +ValueError: The file 'desktop/js/bundles/hue/vendors~hue~notebook~tableBrowser-chunk-f7c8562ecf79bc8f1f16.js' could not be found with . +``` - ... - ValueError: The file 'desktop/js/bundles/hue/vendors~hue~notebook~tableBrowser-chunk-f7c8562ecf79bc8f1f16.js' could not be found with . +rebuild the collection of static files: +```bash +./build/env/bin/hue collectstatic +``` -Re-building the collection of static files should fix it: +### Running the UI tests - ./build/env/bin/hue collectstatic +Tests live next to the file under test. The test filename must end with `.test.ts`, `.test.tsx`, `.test.js` or `.test.jsx` (see `testMatch` in `jest.config.js`): -### Running the UI tests +```text +someFile.ts <- File under test +someFile.test.ts <- File containing tests +``` -The tests are next to the file under test, the filename of the test has to end with `.test.ts` or `.test.js`. +For React components, use [React Testing Library](https://testing-library.com/docs/react-testing-library/intro/) (`@testing-library/react`). - someFile.js <- File under test - someFile.test.ts <- File containing tests +Run all the tests once: -Run all the tests once with: +```bash +npm test +``` - npm test +Run the tests of a specific file once: -Run tests from a specific file once: +```bash +npm test -- foo.test.js +``` - npm test -- foo.test.js +Run the tests in watch mode: -To run the tests in watch mode: +```bash +npm run test-dev +``` - npm run test-dev +In watch mode, Jest detects changes to all files and re-runs the related tests. Press `w` in the console to see the options to target specific files or tests. To update the test snapshots, let the first run of the tests complete and then press `u`. -While in watch mode Jest will detect changes to all files and re-run related tests. There are -also options to target specific files or tests. Press 'w' in the console to see the options. +On some operating systems, like Ubuntu, running the tests via a global Jest can avoid system hangs: -In order to update the test snapshots, let the first checks of tests complete and then -press 'u' in the console to update the snapshots. +```bash +npm install jest --global +``` -Note: on certain OS like Ubuntu, running the tests via a global jest seems to not hang your system +For example: - npm install jest --global +```bash +jest calciteAutocompleteParser.Select.stream.test.js --testPathIgnorePatterns=[] +jest calciteAutocompleteParser --testPathIgnorePatterns=[] +``` -e.g. +Update the snapshots of all tests: - jest calciteAutocompleteParser.Select.stream.test.js --testPathIgnorePatterns=[] - jest calciteAutocompleteParser --testPathIgnorePatterns=[] +```bash +jest --updateSnapshot +``` -How to update snapshot tests: +The tests of some SQL parsers (for example Calcite, Presto or Phoenix) are skipped by default via `testPathIgnorePatterns` in `jest.config.js`. Pass `--testPathIgnorePatterns=[]` as above to run them. - jest --updateSnapshot #### Testing KO Js components @@ -653,281 +738,316 @@ An example of a binding test: ### Coverage -Add the following options: - - ./build/env/bin/pytest --cov=. - -**Note:** Before proceeding, ensure you have installed the pytest-cov package. +The pytest options in `pyproject.toml` already collect the coverage. After `./build/env/bin/pytest`, the reports are in the `reports/` directory (`reports/code-cov/coverage-html` for HTML, `reports/pytest/report.html` for the test report). The `pytest-cov` package is installed by `make test_prep`. -For js run: +For JavaScript, run: - npm run test-coverage - -### Python Linting +```bash +npm run test-coverage +``` -Hue uses [Ruff](https://docs.astral.sh/ruff/) for linting its backend codebase. The Ruff related configs can be found in the [pyproject.toml](https://github.com/cloudera/hue/blob/master/pyproject.toml) file present at the project root. +The report is written to `reports/jest`. -The Github Action CI checks are also leveraging the Ruff linting on branch updates and pull requests. -To manually check for linting violations **on the modified files**, run: +### Python linting - ./build/env/bin/hue runruff check +Hue uses [Ruff](https://docs.astral.sh/ruff/) for linting its backend codebase. The Ruff configuration is in the [pyproject.toml](https://github.com/cloudera/hue/blob/master/pyproject.toml) file at the project root (2-space indentation, 140 characters per line). -The above command will lint all the files modified w.r.t `origin/master` by default and to change the base branch for finding all modified files, simply add the `--diff-branch=` argument like below: +The GitHub Actions CI also runs Ruff on pull requests. - ./build/env/bin/hue runruff check --diff-branch= +To check for linting violations **on the modified files**, run: -Ruff can also autofix most of the flagged violations: +```bash +./build/env/bin/hue runruff check +``` - ./build/env/bin/hue/runruff check --fix +By default, this lints all the Python files modified compared to `origin/master`. To use another base branch, pass `--diff-branch` before the Ruff command: -Some flagged violations can't be autofixed by Ruff and requires manual fixing. Other Ruff supported commands are listed [here](https://docs.astral.sh/ruff/configuration/#full-command-line-interface) +```bash +./build/env/bin/hue runruff --diff-branch= check +``` -### Integration tests +Ruff can autofix most of the flagged violations: -To not fail on integration tests, Hue would need to be configured to point to a live cluster, or you could install the mini cluster (only once) with: +```bash +./build/env/bin/hue runruff check --fix +``` - ./tools/jenkins/jenkins.sh slow +Some violations can't be autofixed and require manual fixing. The other Ruff commands are listed in the [Ruff documentation](https://docs.astral.sh/ruff/configuration/#full-command-line-interface). -Note: the integration tests are going to be ported to a more modern CI with dependencies provided by containers. +### Frontend linting -Those are tagged with `integration` either at the class or method level: +The CI runs the following checks on the frontend code (see `package.json` for the exact commands): - class BeeswaxSampleProvider(object): - integration = True +```bash +npm run prettier # formatting +npm run lint # ESLint +npm run style-lint # Stylelint (SCSS and legacy Less) +``` - @attr('integration') - def test_add_ldap_users_case_sensitivity(self): - if is_live_cluster(): - raise SkipTest('GH-2897: Cannot yet guarantee database is case sensitive') +Use `npm run prettier-fix`, `npm run lint-fix` and `npm run style-lint-fix` to autofix. - ... -## Releasing +### Integration tests -The checklist below details the steps. Then send the release notes to the [Forum](https://discourse.gethue.com/) and https://twitter.com/gethue! +Integration tests need a live cluster or service and are skipped by default (`-m 'not integration'` in `pyproject.toml`). They are tagged with the `integration` pytest marker (or `requires_hadoop` for a live Hadoop cluster) at the class or method level: -Also update https://wikipedia.org/wiki/Hue_(Software). +```python +import pytest -### Version +@pytest.mark.integration +class TestWithLiveCluster: + def test_something(self): + ... +``` -Here is an example of [release commit](https://github.com/cloudera/hue/commit/9de217d6b6). +Point Hue to the live cluster in its configuration before running them. -Update the versions to the next release (current release +1): +Note: the Python integration tests are going to be ported to a more modern CI with dependencies provided by containers. - :100644 100644 4db6d5f... f907d04... M VERSION - :100644 100644 9332f95... 45b28ad... M desktop/libs/librdbms/java/pom.xml - :100644 100644 551f62f... 694f021... M maven/pom.xml - :100644 100644 658d54a... 671ce01... M package.json -How to count the number of commits since the last release: +## Releasing - git log --oneline --since=2020-01-01 | grep 'release' -n -i - git log --oneline -449 > commits.txt +The checklist below details the steps. Then send the release notes to the [Forum](https://discourse.gethue.com/) and the [Hue account on X](https://twitter.com/gethue). - cat commits.txt | sed 's/^\([[:alnum:]]\{10\}\)/\[\1\]\(https:\/\/github.com\/cloudera\/hue\/commit\/\1\)/' | sed 's/^\(.*\)/* \1/' > commits.md +Also update https://wikipedia.org/wiki/Hue_(Software). -And add them and the authors to the release notes: +### Before you start - git log --pretty="%an" | sort | uniq | sed 's/^\(.*\)/* \1/' > authors.txt +* Write access to the repository, to the Docker Hub organization and to the NPM package (for the steps you perform) +* A clean checkout of the commit to release, with the build prerequisites from the [quick start](#requirements) -### Git +### Version -Pushing the release branch: +Update the version to the one being released (for example 4.12.0) in these files: - git push origin HEAD:branch-4.11.0 +| File | Change | +| ---- | ------ | +| `VERSION` | `VERSION="4.12.0"` | +| `NOTICE.txt` | First line: `Cloudera Hue 4.12.0` | +| `maven/pom.xml` | Project version: `4.12.0-SNAPSHOT` | +| `desktop/libs/librdbms/java/pom.xml` | Parent version: `4.12.0-SNAPSHOT` | -Tagging the release: +The `package.json` version is the version of the `gethue` NPM package, which is versioned independently of Hue. Only bump it when you publish a new NPM package. - git tag -a release-4.11.0 -m "release-4.11.0" - git push origin release-4.11.0 +Then add the release notes page `docs/docs-site/content/releases/release-notes-4.12.0.md`, using the previous page as a template. Keep the front matter `weight` ordering consistent with the other release notes. -Draft a new release on https://github.com/cloudera/hue/releases. +How to count the number of commits since the last release: -Publish Github NPM package and Docker images at https://github.com/orgs/cloudera/packages?repo_name=hue. +```bash +git log --oneline --since=2023-01-01 | grep 'release' -n -i +git log --oneline -449 > commits.txt +``` -### Tarball +Then convert them to a Markdown list with links: -Building the tarball release: +```bash +sed 's/^\([[:alnum:]]\{10\}\)/[\1](https:\/\/github.com\/cloudera\/hue\/commit\/\1)/' commits.txt | sed 's/^\(.*\)/* \1/' > commits.md +``` - git checkout -b release-4.11.0 release-4.11.0 - export PYTHON_VER=python3.11 - make prod +And add them and the authors to the release notes: -You might need to upgrade the [Mysqlclient](https://docs.gethue.com/administrator/installation/dependencies/#mysql--mariadb) if seeing: +```bash +git log --pretty="%an" | sort | uniq | sed 's/^\(.*\)/* \1/' > authors.txt +``` - _mysql.c:44:10: fatal error: my_config.h: No such file or directory - 44 | #include "my_config.h" - | ^~~~~~~~~~~~~ +Optionally, add a release blog post in `docs/gethue/content/en/posts`. -Source of the release: https://github.com/cloudera/hue/archive/release-4.11.0.zip +### Git -Push to the CDN: +Commit the changes above and merge them into `master` via a pull request. Then push the release branch (from the release commit): - scp hue-4.11.0.tgz root@cdn.gethue.com:/var/www/cdn.gethue.com/downloads +```bash +git push origin HEAD:branch-4.12.0 +``` -### Docker +Tag the release: -Docker images are at https://hub.docker.com/u/gethue/ +```bash +git tag -a release-4.12.0 -m "release-4.12.0" +git push origin release-4.12.0 +``` - docker build https://github.com/cloudera/hue.git#release-4.11.0 -t gethue/hue:4.11.0 -f tools/docker/hue/Dockerfile - docker tag gethue/hue:4.11.0 gethue/hue:latest - docker images - docker login -u gethue - docker push gethue/hue - docker push gethue/hue:4.11.0 +Draft a new release on [GitHub releases](https://github.com/cloudera/hue/releases). - docker build https://github.com/cloudera/hue.git#release-4.11.0 -t gethue/nginx:4.11.0 -f tools/docker/nginx/Dockerfile; - docker tag gethue/nginx:4.11.0 gethue/nginx:latest - docker push gethue/nginx - docker push gethue/nginx:4.11.0 +### Tarball +Build the tarball from the release tag: -### Kubernetes / Helm +```bash +git checkout release-4.12.0 +export PYTHON_VER=python3.11 +make prod +``` -To build the chart, use the package command from the Helm root directory: +The tarball is generated under `build/release/prod/`. - cd hue/tools/kubernetes/helm/ - helm package hue +You might need to upgrade the [mysqlclient](/administrator/installation/dependencies/#mysql--mariadb) if you see: -Then to publish it to the outside via the current Apache server: +```text +_mysql.c:44:10: fatal error: my_config.h: No such file or directory + 44 | #include "my_config.h" + | ^~~~~~~~~~~~~ +``` - scp hue-1.0.1.tgz root@101.200.100.200:/var/www/helm.gethue.com +The source of the release is available at `https://github.com/cloudera/hue/archive/release-4.12.0.zip`. -Then connect to the server and index the package: +Publish the tarball to the downloads host (for example the CDN of gethue.com): - ssh root@101.200.100.200 - cd /var/www/helm.gethue.com - helm repo index . +```bash +scp hue-4.12.0.tgz @: +``` -### Pypi +### Docker -The Python API and CLI (and soon API) are being transitioned to https://github.com/gethue/compose which host the `compose` and `compose-admin` commands. +The Dockerfiles are in `tools/docker/hue` and `tools/docker/nginx`. Images are published on [Docker Hub](https://hub.docker.com/u/gethue/). -How to update the repo https://pypi.org/project/gethue/: +```bash +docker build https://github.com/cloudera/hue.git#release-4.12.0 -t gethue/hue:4.12.0 -f tools/docker/hue/Dockerfile +docker tag gethue/hue:4.12.0 gethue/hue:latest +docker images +docker login -u gethue +docker push gethue/hue +docker push gethue/hue:4.12.0 - rm -rf dist && python3 -m build && python3 -m twine upload dist/* +docker build https://github.com/cloudera/hue.git#release-4.12.0 -t gethue/nginx:4.12.0 -f tools/docker/nginx/Dockerfile +docker tag gethue/nginx:4.12.0 gethue/nginx:latest +docker push gethue/nginx +docker push gethue/nginx:4.12.0 +``` - python3 -m pip install gethue --upgrade +### Kubernetes and Helm -Note: +The chart is in `tools/kubernetes/helm/hue`. Bump `version` in its `Chart.yaml` if the chart changed, then build the package from the Helm root directory: -A Pypi token will be needed. For doing a test release https://test.pypi.org/project/gethue/ is handy: +```bash +cd tools/kubernetes/helm/ +helm package hue +``` - python3 -m pip install --upgrade build +Then publish it to your Helm repository host and index the package: - rm -rf dist && python3 -m build && python3 -m twine upload --repository testpypi dist/* - python3 -m pip install --index-url https://test.pypi.org/simple/ gethue --upgrade +```bash +scp hue-.tgz @: +ssh @ +cd +helm repo index . +``` ### Documentation -Documentation is currently being auto refreshed every morning of the week and run as a container. - -The manual process otherwise would be to [build it](#documentation-1) and push it to the docs host. +The documentation site is refreshed automatically and runs as a container (see `tools/docker/documentation`). -The resources like images go to the CDN: +Otherwise, [build it](#documentation-1) and push it to the docs host. - scp before_raz_adls.png after_raz_adls.png root@104.248.178.223:/var/www/cdn.gethue.com/uploads/2021/09/ +Resources like images go to the CDN: -### NPM registry +```bash +scp @: +``` -To publish gethue to the [NPM registry](https://www.npmjs.com/package/gethue), the following command would have to be run. - npm run publish-gethue +### NPM registry -## Building +To publish the `gethue` package to the [NPM registry](https://www.npmjs.com/package/gethue), bump the `version` in `package.json` and run: -### Custom Webpack Config +```bash +npm run publish-gethue +``` -You can make webpack build with custom configuration files by setting HUE_WEBPACK_CONFIG environment variable. +This regenerates the SQL parsers (`make sql-all-parsers`), builds the package with `webpack.config.npm.js` into `npm_dist` and publishes it. You need to be logged in to NPM (`npm login`). - HUE_WEBPACK_CONFIG="webpack.config.custom.js" make apps +## Building ### Dev Docker Try basic changes [in 3 minutes](https://gethue.com/quick-start-a-hue-development-environment-in-3-minutes-with-docker/) without compiling Hue locally hence avoiding the setting up of [dependencies](/developer/): - git clone https://github.com/cloudera/hue.git - cd hue - cp desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini +```bash +git clone https://github.com/cloudera/hue.git +cd hue +cp desktop/conf/pseudo-distributed.ini.tmpl desktop/conf/pseudo-distributed.ini +``` -Then edit the `[[database]]` section to specify a proper database, here MySql: +Then edit the `[[database]]` section to specify a proper database, here MySQL: - host=127.0.0.1 # Don't use 'localhost' if Docker - engine=mysql - user=hue - password=hue - name=huedb +```ini +host=127.0.0.1 # Don't use 'localhost' if Docker +engine=mysql +user=hue +password=hue +name=huedb +``` Then map the local Hue source code into the running container (so that local edits are seen in the running Hue): - sudo docker run -it -v $PWD/apps:/usr/share/hue/apps -v $PWD/desktop:/usr/share/hue/desktop -v $PWD/desktop/conf/pseudo-distributed.ini:/usr/share/hue/desktop/conf/z-hue.ini --network="host" gethue/hue +```bash +sudo docker run -it -v $PWD/apps:/usr/share/hue/apps -v $PWD/desktop:/usr/share/hue/desktop -v $PWD/desktop/conf/pseudo-distributed.ini:/usr/share/hue/desktop/conf/z-hue.ini --network="host" gethue/hue +``` -Note: code updates won’t be seen after the Docker container runs. For this Hue would need to be started in dev server mode by replacing the [line](https://github.com/cloudera/hue/blob/master/tools/docker/hue/startup.sh#L5) by +Note: code updates won’t be seen after the Docker container runs. For this, Hue needs to be started in dev server mode by replacing the `supervisor` line of [`startup.sh`](https://github.com/cloudera/hue/blob/master/tools/docker/hue/startup.sh) by: - ./build/env/bin/hue runserver 0.0.0.0:8888 +```bash +./build/env/bin/hue runserver 0.0.0.0:8888 +``` -and recompiling the Docker image. It will then auto-restart on Python code changes. For JavaScript, those would need to be [compiled](/developer/development/#javascript). +and recompiling the Docker image. It will then auto-restart on Python code changes. For JavaScript, those would need to be [compiled](/developer/development/#javascript-and-typescript). ### Documentation -Install [Hugo](https://gohugo.io/getting-started/quick-start/). Each page has a link to its own source file in the top right corner. +Install [Hugo](https://gohugo.io/getting-started/quick-start/). The `Docs Lint` CI job builds the site with Hugo 0.69.0, so use that version to check your changes locally (the theme does not render with recent Hugo versions). Each page has a link to its own source file in the top right corner. Build the [source](https://github.com/cloudera/hue/tree/master/docs/docs-site) and see live changes: - cd docs/docs-site - - hugo serve - -Check for links not working (e.g. returning a 404) with muffet, a fast link checker crawler. It recommended to use the [check links script](https://github.com/cloudera/hue/blob/master/tools/ci/check_for_website_dead_links.sh). - -The posts [manual](https://gethue.com/easily-checking-for-deadlinks-on-docs-gethue-com/) and [continuous integration](https://gethue.com/easily-checking-for-deadlinks-on-docs-gethue-com/) contain more information about it. - -And then to build the static site just do: +```bash +cd docs/docs-site +hugo serve +``` - hugo +The [Docs Lint](https://github.com/cloudera/hue/blob/master/.github/workflows/docs_lint.yml) workflow serves the site and checks it for dead links with [Lychee](https://github.com/lycheeverse/lychee) on pull requests that change `docs/docs-site`. You can run the same check locally against `http://localhost:1313/`. -and grab the `public` directory. +To build the static site, run `hugo` in `docs/docs-site` (or `make doc` from the Hue root) and grab the `public` directory. ### gethue.com Blog & Website -Like for the [Documentation](#documentation-1) install hugo. The content for each language is in its [own directory](https://github.com/cloudera/hue/tree/master/docs/gethue/content). +Like for the [Documentation](#documentation-1), install Hugo. The content for each language is in its [own directory](https://github.com/cloudera/hue/tree/master/docs/gethue/content). Blog posts are located in [docs/gethue/content/en/posts](https://github.com/cloudera/hue/tree/master/docs/gethue/content/en/posts). Here is an example of the source of a [release post](https://github.com/cloudera/hue/commit/fcc0078a4ba59e450a2e081ad0cc27e482a8703b#diff-9b1261f162231be68d3c30329e73b458c874858c0d685cf0c213fdace3309679). Build it and see live changes: - cd docs/gethue - - hugo serve +```bash +cd docs/gethue +hugo serve +``` -Will automatically start one server for each language domain. +This automatically starts one server for each language domain. gethue.com refresh is currently manually kicked but this will be automated soon. -### SQL Autocomplete - -Install a patched jison: +### SQL autocomplete - git clone https://github.com/JohanAhlen/jison - cd jison - npm install -g . +The parsers are generated with a patched [jison](https://github.com/JohanAhlen/jison), which is installed as a dependency of `tools/jison` (no global install is needed). To regenerate the generic, Hive and Impala parsers, run: -Then run: +```bash +make sql-all-parsers +``` - make sql-all-parsers +See [SQL parsers](#sql-parsers) for the other dialects. -### Ace Editor +### Ace editor -After modifying files under tools/ace-editor run the following to build ace.js +After modifying files under `tools/ace-editor`, run the following to build `ace.js`: - npm install - make ace +```bash +npm install +make ace +``` ### Language references -The tools for generating the embedded language reference manuals can be found under `hue/tools/sql-docs/` +The tools for generating the embedded language reference manuals are in `tools/sql-docs/`. #### Hive The Hive documentation is generated directly from the Hive wiki by using an exported epub file. @@ -937,7 +1057,9 @@ The Hive documentation is generated directly from the Hive wiki by using an expo 3. Click 'Export to EPUB' 4. In the Hue folder run: - node tools/sql-docs/hiveExtractor.js --epub /path/to/epub/file + ```bash + node tools/sql-docs/hiveExtractor.js --epub /path/to/epub/file + ``` #### Impala The Impala documentation is generated from the ditamap files in the Impala GitHub repo. @@ -945,36 +1067,44 @@ The Impala documentation is generated from the ditamap files in the Impala GitHu 1. Clone the Impala repo next to hue from https://github.com/apache/impala 2. In the Hue folder run: - node tools/sql-docs/docExtractor.js -c hue -f ../impala/docs/ -d impala_langref.ditamap,impala_keydefs.ditamap,impala.ditamap -o desktop/core/src/desktop/static/desktop/docs/impala/ -m desktop/core/src/desktop/templates/impala_doc_index.mako + ```bash + node tools/sql-docs/docExtractor.js -c hue -f ../impala/docs/ -d impala_langref.ditamap,impala_keydefs.ditamap,impala.ditamap -o desktop/core/src/desktop/static/desktop/docs/impala/ -m desktop/core/src/desktop/templates/impala_doc_index.mako + ``` ### Internationalization -How to update all the messages and compile them: +How to compile all the messages: - make locales +```bash +make locales +``` -How to update and compile the messages of one app: +How to compile the messages of one app: - cd apps/beeswax - make compile-locale +```bash +cd apps/beeswax +make compile-locale +``` How to create a new locale for an app: - cd $APP_ROOT/src/$APP_NAME/locale - $HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr +```bash +cd $APP_ROOT/src/$APP_NAME/locale +$HUE_ROOT/build/env/bin/pybabel init -D django -i en_US.pot -d . -l fr +``` -## SQL Parsers +## SQL parsers -The parsers are the flagship part of Hue and power extremely advanced autocompletes and other [SQL functionalities](/user/querying/#autocomplete). They are running on the client side and comes with just a few megabytes of JavaScript that are cached by the browser. This provides a very reactive experience to the end user and allows to [import them](#reusing-a-parser-in-your-project) as classic JavaScript modules for your own development needs. +The parsers are the flagship part of Hue and power extremely advanced autocompletes and other [SQL functionalities](/user/querying/#autocomplete). They run on the client side and come with just a few megabytes of JavaScript that are cached by the browser. This provides a very reactive experience to the end user and let you [import them](#reusing-a-parser-in-your-project) as classic JavaScript modules for your own development needs. While the dynamic content like the list of tables, columns is obviously fetched via [remote endpoints](/administrator/configuration/connectors/), all the SQL knowledge of the statements is available. The main dialects are: -* Apache Hive -* Apache Impala -* Presto -* Apache Calcite +* Apache Hive +* Apache Impala +* Presto +* Apache Calcite But there are more! See all the currently shipped [SQL dialects](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql). @@ -990,11 +1120,11 @@ This guide takes you through the steps necessary to create an autocompleter for - Adding `SHOW` syntax to Flink SQL [GH-1399](https://github.com/cloudera/hue/issues/1399) - Adding a Dask SQL parser and connector [GH-1480](https://github.com/cloudera/hue/issues/1480) -### Parser Theory +### Parser theory There are several parsers in Hue already (e.g. one for Impala, one for Hive..) and a generic SQL that is used for other dialects. The parsers are written using a [bison](https://www.gnu.org/software/bison/) grammar and are generated with [jison](https://github.com/zaach/jison). They are 100% Javascript and live on the client side, this gives the performance of a desktop editor in your browser. -Building a dedicated work is more effort but it then allows a very rich end user experience, e.g.: +Building a dedicated parser is more effort but it then allows a very rich end user experience, e.g.: * Handle invalid/incomplete queries and propose suggestions/fixes * date_column = @@ -1039,7 +1169,8 @@ Here’s a list of some of the different types of suggestions the parser can ide suggestDatabases suggestFilters suggestFunctions - suggestGroupBys suggestHdfs + suggestGroupBys + suggestHdfs suggestIdentifiers suggestJoinConditions suggestJoins @@ -1049,45 +1180,45 @@ Here’s a list of some of the different types of suggestions the parser can ide suggestTables suggestValues -Parsers are generated and added to the repository using the command generateParsers.js under tools/jison/. To for instance generate all the Impala parsers you would run the following command in the hue folder: +Parsers are generated and added to the repository with `generateParsers.js` under `tools/jison/`. For instance, to generate all the Impala parsers, run the following in the Hue folder: - cd tools/jison - node generateParsers.js impala +```bash +cd tools/jison +npm install +node generateParsers.js impala +``` -In reality two parsers are generated per dialect, one for syntax and one for autocomplete. The syntax parsers is a subset of the autocomplete parser with no error recovery and without the autocomplete specific grammar. +The arguments are parser names (for example `hiveAutocompleteParser`), dialect name prefixes (for example `hive`) or `all`. Each dialect folder under `desktop/core/src/desktop/js/parse/sql/` that contains a `jison/structure.json` file is picked up automatically. -All the jison grammar files can be found [here](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/jison/sql) and the generated parsers are also committed together with their tests [here](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql). +In reality two parsers are generated per dialect, one for syntax and one for autocomplete. The syntax parser is a subset of the autocomplete parser with no error recovery and without the autocomplete specific grammar. -Parsers are sharing a maximum of the common syntax via some modules so that it is easy to improve the specificness of any of them while not starting from sratch. +The jison grammar files of each dialect are in its `jison` folder, e.g. [parse/sql/hive/jison](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql/hive/jison). The generated parsers are committed next to them in [parse/sql](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql), together with their tests. -e.g. in [structure.json](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/js/parse/jison/sql/hive/structure.json): +Parsers share as much of the common syntax as possible via modules, so that it is easy to improve the specificity of any of them without starting from scratch. - { - "lexer": "sql.jisonlex", - "autocomplete": [ - "../generic/autocomplete_header.jison", - "abort/abort_transactions.jison", - "common/table_constraint.jison", - "alter/alter_common.jison", - "alter/alter_database.jison", - "alter/alter_index.jison", - "alter/alter_materialized_view.jison", - "alter/alter_table.jison", - "alter/alter_view.jison", - "analyze/analyze_table.jison", - ... - ], - "syntax": [ - "../generic/syntax_header.jison", - "abort/abort_transactions.jison", - "common/table_constraint.jison", - "alter/alter_common.jison", - "alter/alter_database.jison", - "alter/alter_index.jison", - "alter/alter_materialized_view.jison", - ... - ] - } +e.g. in [structure.json](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/js/parse/sql/hive/jison/structure.json): + +```json +{ + "lexer": "sql.jisonlex", + "autocomplete": [ + "../../generic/jison/autocomplete_header.jison", + "abort/abort_transactions.jison", + "common/table_constraint.jison", + "alter/alter_common.jison", + "alter/alter_connector.jison", + "alter/alter_database.jison", + ... + ], + "syntax": [ + "../../generic/jison/syntax_header.jison", + "abort/abort_transactions.jison", + "common/table_constraint.jison", + "alter/alter_common.jison", + ... + ] +} +``` #### The grammar @@ -1140,39 +1271,55 @@ The goal is to create from scratch a new parser for the PostgreSQL database. #### Prerequisites -Make sure you have [jison](/developer/development/#sql-autocomplete) installed and a [development](/administrator/installation/dependencies/) Hue. Then configure a [PostgreSQL interpreter](/administrator/configuration/connectors/#postgresql). +Make sure you have a [development](/administrator/installation/dependencies/) Hue and ran `npm install` in the Hue folder and in `tools/jison` (this installs the patched [jison](#sql-autocomplete)). Then configure a [PostgreSQL interpreter](/administrator/configuration/connectors/#postgresql). In the Hue folder: - ./build/env/bin/pip install psycopg2-binary +```bash +./build/env/bin/pip install psycopg2-binary +``` -and edit your hue config desktop/conf/pseudo-distributed.ini to contain: +and edit your Hue config `desktop/conf/pseudo-distributed.ini` to contain: - [notebook] - [[interpreters]] - [[[postgresql]]] - name = postgresql - interface=sqlalchemy - options='{"url": "postgresql://hue:hue@localhost:31335/hue"}' +```ini +[notebook] +[[interpreters]] +[[[postgresql]]] +name = postgresql +interface=sqlalchemy +options='{"url": "postgresql://hue:hue@localhost:31335/hue"}' +``` -Our generateParsers tool can take an existing dialect and setup the source code for a new parsers based on that. +The parser generator discovers every dialect folder under `desktop/core/src/desktop/js/parse/sql/` that contains a `jison/structure.json` file. Create the new dialect by copying the `generic` one, in the Hue folder: -In the hue folder run: +```bash +cd desktop/core/src/desktop/js/parse/sql +cp -r generic postgresql +cd postgresql +rm genericAutocompleteParser*.js genericAutocompleteParser.test.ts genericSyntaxParser*.js +mkdir test +``` - cd tools/jison - npm install - node generateParsers.js -new generic postgresql +Then register the new dialect in `desktop/core/src/desktop/js/parse/sql/sqlParserRepository.ts` by adding a `postgresql` entry next to the `hive` one in both `AUTOCOMPLETE_MODULES` (importing `./postgresql/postgresqlAutocompleteParser`) and `SYNTAX_MODULES` (importing `./postgresql/postgresqlSyntaxParser`). -After the -new argument you specify an existing dialect to clone first and then the name of the new parser. +Generate the new `postgresqlAutocompleteParser.js` and `postgresqlSyntaxParser.js`: -Once executed the tool has cloned the generic parser with tests and generated a new postgresql parsers. The jison files can be found under `desktop/core/src/desktop/js/parse/jison/sql/postgresql/` and the testscan be found in `desktop/core/src/desktop/js/parse/sql/postgresql/test`. +```bash +cd tools/jison +npm install +node generateParsers.js postgresql +``` -To regenerate the parsers after changes to the jison files run: +The jison files are under `desktop/core/src/desktop/js/parse/sql/postgresql/jison/` and the tests go in `desktop/core/src/desktop/js/parse/sql/postgresql/test`. - cd tools/jison - node generateParsers.js postgresql +To regenerate the parsers after changes to the jison files, run the same command again: + +```bash +cd tools/jison +node generateParsers.js postgresql +``` -The tool will report any problems with the grammar. Note that it might still generate a parser if the grammar isn’t ambiguous but it’s likely that there will be test failures. +The tool reports any problems with the grammar. Note that it might still generate a parser if the grammar isn’t ambiguous, but it’s likely that there will be test failures. #### Extending the grammar @@ -1180,7 +1327,35 @@ This gives you an idea on how to add custom syntax to the newly generated postgr REINDEX { INDEX | TABLE | DATABASE | SYSTEM } name [ FORCE ] -We’ll start by adding a test, in `postgresqlAutocompleteParser.test.js` in the test folder inside the main describe function before the first `it('should...`: +We’ll start by adding a test. Create `postgresqlAutocompleteParser.test.js` in the test folder, modeled on the [Hive tests](https://github.com/cloudera/hue/tree/master/desktop/core/src/desktop/js/parse/sql/hive/test): + +```js +import postgresqlAutocompleteParser from '../postgresqlAutocompleteParser'; + +describe('postgresqlAutocompleteParser.js', () => { + beforeAll(() => { + postgresqlAutocompleteParser.yy.parseError = function (msg) { + throw Error(msg); + }; + }); + + const assertAutoComplete = testDefinition => { + const debug = false; + + expect( + postgresqlAutocompleteParser.parseSql( + testDefinition.beforeCursor, + testDefinition.afterCursor, + debug + ) + ).toEqualDefinition(testDefinition); + }; + + // Add the tests below here +}); +``` + +Then add the following tests inside the main `describe` function: describe('REINDEX', () => { it('should handle "REINDEX TABLE foo FORCE; |"', () => { @@ -1277,16 +1452,18 @@ We also want the autocompleter to suggest the keyword REINDEX when the user hasn }); }); -For this to pass we need to add REINDEX to the list of DDL and DML keywords in the file `sqlParseSupport.js` next to the generated parser (`desktop/core/src/desktop/js/parse/sql/postgresql/sqlParseSupport.js/`). Find the function `parser.suggestDdlAndDmlKeywords` and add ‘REINDEX’ to the keywords array. Now run `npm run test -- postgresqlAutocompleteParser.test.js` and the three tests should pass. +For this to pass we need to add REINDEX to the list of DDL and DML keywords in the file `sqlParseSupport.js` next to the generated parser (`desktop/core/src/desktop/js/parse/sql/postgresql/sqlParseSupport.js`). Find the function `parser.suggestDdlAndDmlKeywords` and add ‘REINDEX’ to the keywords array. Now run `npm run test -- postgresqlAutocompleteParser.test.js` and the three tests should pass. Before you continue further, note that in this case there will be two new failing tests where the keyword ‘REINDEX’ has to be added. In order to use the newly generated parsers we have to add them to the webpack bundles: - npm run webpack - npm run webpack-workers +```bash +npm run webpack +npm run webpack-workers +``` -While developing it will speed up if the webpack bundling runs in the background, for this open two terminal sessions and run `npm run dev` in one and `npm run dev-workers` in the other. It will then monitor changes to the files and build a lot quicker. +While developing, it is faster to run the webpack bundling in the background. Open two terminal sessions and run `npm run dev` in one and `npm run dev-workers` in the other. They monitor changes to the files and rebuild a lot quicker. After the bundling you can now test it directly in the editor! @@ -1335,7 +1512,7 @@ To add a new dialect, it is recommended to copy the two files of the closest mod Tip: inheritance of modes is supported by Ace, which make it handy for avoiding potential duplications. -In the Editor, the mapping between Ace's modes and the type of snippets is happening in [editor_components.mako](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/templates/editor_components.mako#L2118). +In the Editor, the mapping between Ace's modes and the type of snippets is happening in [editor2.mako](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/templates/editor2.mako). In the KSQL case we have: @@ -1369,7 +1546,7 @@ Connectors provide pluggable integration to any external data service so that an #### SqlAlchemy -[SqlAlchemy](https://www.sqlalchemy.org) is the prefered way if the Hive API is not supported by the database. The core implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and relies on each respective SqlAlchemy dialect. +[SqlAlchemy](https://www.sqlalchemy.org) is the preferred way if the Hive API is not supported by the database. The core implementation is in [`sql_alchemy.py`](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/sql_alchemy.py) and relies on each respective SqlAlchemy dialect. #### Hive Interface @@ -1386,11 +1563,13 @@ If the built-in HiveServer2 (Hive, Impala, Spark SQL), SqlAlchemy (MySQL, Postgr * [Solr SQL](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/solr.py) * [JDBC](https://github.com/cloudera/hue/blob/master/desktop/libs/notebook/src/notebook/connectors/jdbc.py) -The JDBC API relies on a small JDBC proxy running next to the Hue API. By default it won't be built without setting the `BUILD_DB_PROXY` flag, e.g.: +The JDBC API relies on a small JDBC proxy running next to the Hue API. By default it is not built unless you set the `BUILD_DB_PROXY` variable, e.g.: - export BUILD_DB_PROXY=true make install +```bash +BUILD_DB_PROXY=true make install +``` -**Note** In the long term, SqlAlchemy is prefered as more "Python native". +**Note** In the long term, SqlAlchemy is preferred as more "Python native". #### Potential connectors @@ -1439,10 +1618,6 @@ Various storage systems can be interacted with. The [`fsmanager.py`](https://git With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py), the HBase browser could be compatible with Apache Kudu or Google Big Table. -#### Potential connectors - -* Google Cloud Storage is currently a work in progress with [HUE-8978](https://issues.cloudera.org/browse/HUE-8978) - ### Jobs #### Apache Spark / Livy @@ -1457,11 +1632,11 @@ Based on the [Livy REST API](/administrator/configuration/connectors/#apache-spa #### Schedulers -Currently only Apache Oozie is supported for your Datawarehouse, but the API is getting generic with [HUE-3797](https://issues.cloudera.org/browse/HUE-3797). +Currently only Apache Oozie is supported for your Datawarehouse, but the scheduler API is generic so that other schedulers can be added. #### Potential connectors -* Elastic Search: a connector similar to Solr for Searching [HUE-7828](https://issues.cloudera.org/browse/HUE-7828). SQL querying is already supported. +* Elastic Search: a connector similar to Solr for searching. SQL querying is already supported. * [Livy Browser API](https://github.com/cloudera/hue/blob/master/apps/jobbrowser/src/jobbrowser/apis/livy_api.py) * [Celery API](https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/lib/scheduler/lib/beat.py) * Apache Hive native support of query scheduling [HIVE-21884](https://issues.apache.org/jira/browse/HIVE-21884) diff --git a/docs/docs-site/content/quickstart/_index.md b/docs/docs-site/content/quickstart/_index.md index 72619756dce..6ec006d0d7e 100644 --- a/docs/docs-site/content/quickstart/_index.md +++ b/docs/docs-site/content/quickstart/_index.md @@ -1,32 +1,41 @@ --- -title: "Quick Start" +title: "Quick start" date: 2019-03-13T18:28:08-07:00 draft: false weight: 1 --- -## Get +## Get Hue -Via [Docker](/administrator/installation/cloud/#docker): +Start Hue with [Docker](/administrator/installation/cloud/#docker): - docker run -it -p 8888:8888 gethue/hue:latest +```bash +docker run -it -p 8888:8888 gethue/hue:latest +``` -Or [Kubernetes](/administrator/installation/cloud/#kubernetes): +Open [http://localhost:8888](http://localhost:8888). The first user who logs in becomes the superuser. - helm repo add gethue https://helm.gethue.com - helm repo update - helm install hue gethue/hue +Or start Hue on [Kubernetes](/administrator/installation/cloud/#kubernetes) with Helm: -## Configure +```bash +helm repo add gethue https://helm.gethue.com +helm repo update +helm install hue gethue/hue +``` -Then [configure the server](/administrator/configuration/) and point connectors to your [Databases, Warehouses, Storages](/administrator/configuration/connectors/). +The command prints how to reach the Hue service. To build Hue from source instead, see the [installation guide](/administrator/installation/). -## Use +> **Note:** The Docker image uses an embedded SQLite database by default. Connect Hue to [MySQL or PostgreSQL](/administrator/administration/database/) before real use. -Here are some tutorials on how to: +## Configure Hue -* [Create and Query a Customer 360](http://gethue.com/self-service-bi-doing-a-customer-360-by-querying-and-joining-salesforce-marketing-and-log-datasets/) -* [Upload data and turn it into a SQL table](http://gethue.com/querying-exploring-the-instacart-dataset-part-1-ingesting-the-data/) -* [Ingest and analyse bike data](https://docs.cloudera.com/runtime/7.0.1/using-hue/topics/hue-using.html) +[Configure the server](/administrator/configuration/) and point the connectors to your [databases, warehouses, and storages](/administrator/configuration/connectors/). -For development please check how to [install](/administrator/installation/dependencies/) or [contribute](/developer/). +## Use Hue + +These tutorials show how to: + +* [Create and query a Customer 360](https://gethue.com/self-service-bi-doing-a-customer-360-by-querying-and-joining-salesforce-marketing-and-log-datasets/) +* [Upload data and turn it into a SQL table](https://gethue.com/querying-exploring-the-instacart-dataset-part-1-ingesting-the-data/) + +To build Hue from source, see the [dependencies](/administrator/installation/dependencies/). To contribute, see the [developer guide](/developer/). diff --git a/docs/docs-site/content/user/_index.md b/docs/docs-site/content/user/_index.md index a4b4f4ce729..7d250d4092e 100644 --- a/docs/docs-site/content/user/_index.md +++ b/docs/docs-site/content/user/_index.md @@ -8,26 +8,20 @@ pre = "2. " This section describes the main functionalities from an end user point of view. -Hue consists in a single page interface that allow the users to perform data -analysis without losing any context. The goal is to promote self service and stay simple like Excel -so that 80% of the users can find, explore and query data and become more data driven. +Hue is a single-page interface that lets you find, explore, and query data without losing your context. -Please refer to [connector section](/administrator/configuration/connectors/#databases) for the list of the most popular Database/Data Warehouse supported. +See the [connectors page](/administrator/configuration/connectors/#databases) for the list of supported databases and data warehouses. -**1. Find or import your data** +## Typical workflow -Use the left metadata assists to browse your existing data without losing your editor. The top search will look through your saved queries and matching tables, columns and databases of connected [Data Catalogs](/user/browsing/#data-catalog). Objects can be tagged for a quick retrieval or assigning a more “humane” name. If the data does not exist yet, just drag & drop it to trigger the Create Table wizard and import it in just two steps. +1. **Find or import your data.** Use the left assist panel to browse your data without leaving the Editor. The top search looks through your saved queries and the tables, columns, and databases of connected [data catalogs](/user/browsing/#data-catalogs). If the data does not exist yet, [import it](/user/browsing/#importing-data). +2. **Query your data.** The [Editor](/user/querying/#editor) autocompletes SQL, highlights syntax errors, and shows inline documentation. The right assist panel shows table previews and the language reference. Refine your results, then download them as CSV or XLS, or export them to a file system or a table. -**2. Query your data** - -When you found your data, the [Editor's](/user/querying/#editor) autocomplete is extremely powerful and will highlight any syntax or logical error and provide risk alerts as well as inline documentation. The right assistant provides quick previews of the datasets, which columns or JOINs are popular and recommendations on how to type optimized queries. After the querying, refine your results before exporting to S3/HDFS/ADLS or downloaded as CSV/Excel. - -**3. Applications** - -* Editor: The goal of Hue's Editor is to make data querying easy and productive. It focuses on SQL but also supports job submissions. It comes with an intelligent autocomplete, search & tagging of data and query assistance. -* Browsers: Hue's Browsers let you easily search, glance and perform actions on data or jobs in Cloud or on premise clusters. -* Dashboard: Dashboards are an interactive way to explore your data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks. -* Scheduler: The application lets you build workflows and then schedule them to run regularly automatically. A monitoring interface shows the progress, logs and allow actions like pausing or stopping jobs. +## Applications +* **Editor**: Write and run SQL queries with autocomplete and query assistance. +* **Browsers**: Search, preview, and act on tables, files, and jobs. +* **Dashboard**: Explore your data interactively with drag and drop, without programming. Enabled by default. +* **Scheduler**: Build workflows and schedule them to run regularly. It requires the Oozie app to be configured by your administrator. Each app of Hue can be extended to support your own languages or apps as detailed in the [developer guide](/developer/). diff --git a/docs/docs-site/content/user/browsing/_index.md b/docs/docs-site/content/user/browsing/_index.md index 2541b7cbf14..bc29dded719 100644 --- a/docs/docs-site/content/user/browsing/_index.md +++ b/docs/docs-site/content/user/browsing/_index.md @@ -5,18 +5,17 @@ draft: false weight: 3 --- -Browsers power the Data Catalog. They let you easily search, glance, import datasets or jobs. +Browsers let you search, explore and import datasets, and monitor jobs. -The browsers can be "enriched" with external [catalog/metadata services](/administrator/configuration/connectors/#storage/). +You can enrich the browsers with an external catalog or metadata service. See the [connectors configuration](/administrator/configuration/connectors/). ## Catalogs ### Tables -The Table Browser enables you to manage the databases, tables, and partitions of the metastore shared by the Hive and Impala. You can perform the following operations: - -- Search and display metadata like tags and additional description +The Table Browser lets you manage the databases, tables, and partitions of the metastore shared by Hive and Impala. You can perform the following operations: +- Search and display metadata like tags and descriptions - Databases - Select a database - Create a database @@ -30,49 +29,49 @@ The Table Browser enables you to manage the databases, tables, and partitions of - Import data into a table - [Filter, Sort and Browse Partitions](http://gethue.com/filter-sort-browse-hive-partitions-with-hues-metastore/) -### Data Catalogs +### Data catalogs -Before typing any query to get insights, users need to find and explore the correct datasets. It is accessible from the top bar of the interface and offers free text search of SQL tables, columns, tags and saved queries. This is particularly useful for quickly looking up a table among thousands or finding existing queries already analyzing a certain dataset. +Before you write a query, you need to find the right dataset. The Data Catalog search is in the top bar of the interface. It offers free text search of SQL tables, columns, tags and saved queries. This is particularly useful for quickly looking up a table among thousands or finding existing queries already analyzing a certain dataset. -Existing tags, descriptions and indexed objects show up automatically, any additional tags you add appear back in metadata server, and the familiar metadata server search syntax is supported. +Existing tags, descriptions and indexed objects show up automatically. Tags that you add are also sent back to the metadata server, and the metadata server search syntax is supported. ![Data Catalog top search](https://cdn.gethue.com/uploads/2018/04/blog_top_search_.png) -Searching all the available queries or data in the cluster +Searching all the available queries and data in the cluster. ![Data Catalog tags](https://cdn.gethue.com/uploads/2018/04/blog_tag_listing.png) -Listing the possible tags to filter on. This also works for ‘types’. +Listing the tags to filter on. This also works for types. #### Unification of metadata -The list of tables and their columns is displayed in multiple part of the interface. This data is pretty costly to fetch and comes from different sources. In this new version, the information is now cached and reused by all the Hue components. As the sources are diverse, e.g. Apache Hive, Apache Atlas those are stored into a single object, so that it is easier and faster to display without caring about the underlying technical details. +Hue shows the list of tables and their columns in multiple parts of the interface. This data comes from different sources, such as Apache Hive and Apache Atlas. Hue caches it in a single object and reuses it in all components, so it displays faster. -In addition to editing the tags of any SQL objects like tables, views, columns… which has been available since version one, table descriptions can now also be edited. This allows a self service documentation of the metadata by the end users, which was not possible until now as directly editing Hive comments require some admin Sentry privileges which are not granted to regular users in a secure cluster. +You can edit the tags of any SQL object, like tables, views and columns. You can also edit table descriptions. This lets you document the metadata yourself, without the admin privileges that editing Hive comments directly requires on a secure cluster. ![Data Catalog](https://cdn.gethue.com/uploads/2018/04/blog_metadata.png) #### Search -By default, only tables and views are returned. To search for columns, partitions, databases use the ‘type:' filter. +By default, only tables and views are returned. To search for columns, partitions or databases, use the `type:` filter. -Example of searches: +Examples of searches: -Atlas +**Apache Atlas** -* sample → Any table or Hue document with prefix ‘sample’ will be returned -* type:database → List all databases on this cluster -* type:table → List all tables on this cluster -* type:field name → List tables with field(column): ‘name’ -* ‘tag:classification_testdb5’ or ‘classification:classification_testdb5’→ List entities with classification ‘classification_testdb5’ -* owner:admin → List all tables owned by ‘admin’ user +* `sample`: any table or Hue document with the prefix `sample` +* `type:database`: all databases on this cluster +* `type:table`: all tables on this cluster +* `type:field name`: tables with a field (column) named `name` +* `tag:classification_testdb5` or `classification:classification_testdb5`: entities with the classification `classification_testdb5` +* `owner:admin`: all tables owned by the `admin` user -Navigator +**Cloudera Navigator** (deprecated in favor of the `[[catalog]]` configuration) -* table:customer → Find the customer table -* table:tax* tags:finance → List all the tables starting with tax and tagged with ‘finance' -* owner:admin type:field usage → List all the fields created by the admin user that matches the usage string -* parentPath:"/default/web_logs" type:FIELD originalName:b* → List all the columns starting with `b` of the table `web_logs` in the database `default`. +* `table:customer`: the `customer` table +* `table:tax* tags:finance`: all tables starting with `tax` and tagged with `finance` +* `owner:admin type:field usage`: all fields created by the `admin` user that match `usage` +* `parentPath:"/default/web_logs" type:FIELD originalName:b*`: all columns starting with `b` of the table `web_logs` in the database `default` ![Data Catalog Search](https://cdn.gethue.com/uploads/2019/06/SearchWithType_field_name.png) @@ -80,104 +79,123 @@ Learn more on the [Search](http://gethue.com/realtime-catalog-search-with-hue-an #### Tagging -In addition, you can also now tag objects with names to better categorize them and group them to different projects. These tags are searchable, expediting the exploration process through easier, more intuitive discovery. +You can tag objects with names to categorize them and group them by project. Tags are searchable, which makes discovery easier. ![Data Catalog](https://cdn.gethue.com/uploads/2016/04/tags.png) -### Importing Data - -The goal of the importer is to allow ad-hoc queries on data not yet in the clusters and simplifies self-service analytics. +### Importing data -If you want to import your own data instead of installing the sample -tables, open the importer from the left menu or from the little `+` in the left assist. +The Importer lets you query data that is not yet in the cluster. It creates a table from a file in a few steps. +To import your own data, open the importer from the left menu or from the `+` in the left assist panel. To learn more, watch the video on [Data Import Wizard](http://gethue.com/import-data-to-be-queried-via-the-self-service-drag-drop-create-table-wizard/). -**Note** Files can be dragged & dropped, selected from HDFS or S3 (if configured), and their formats are automatically detected. The wizard also assists when performing advanced functionalities like table partitioning, Kudu tables, and nested types. +> **Note:** You can select files from the configured storage (for example HDFS or S3) or upload a small file. Hue detects the file format automatically. The wizard also helps with table partitioning, Kudu tables and nested types. -#### CSV file +#### New importer + +When the administrator sets `enable_new_importer=true` in the `[desktop]` section of `hue.ini` (the default is `false`), a new importer is available. It adds a second entry named **Import**, which opens the page at `/newimporter`. The existing importer stays available. + +To import a file with the new importer: -Any small CSV file can be ingested into a new index in a few clicks. +1. Open the importer and select a source. Depending on your configuration, the choices are **Upload from File**, **HDFS**, **Amazon S3**, **Azure Storage**, **Google Storage** and **Ozone**. +2. Select the file. Hue opens the **Preview** tab with a preview of the data. +3. In the **Preview** tab, choose the database, the table name, the engine and the compute. Review and edit the column names, types and comments. +4. In the **Settings** tab, check the file format: file type (CSV, JSON or Excel), whether the file has a header, the field separator and the quote character. +5. Optional: add partitions in the **Partitions** tab. +6. Click **Finish Import**. + +The new table is created in the selected database. + +> **Note:** Files above the supported size limit can't be uploaded in the importer. Use a file browser to upload them instead. + +#### CSV file -#### Relational Databases +You can ingest any small CSV file into a new index in a few clicks. -Import data from relational databases to HDFS file or Hive table using Apache Sqoop. It enables to bring large amount of data into the cluster in just few clicks via interactive UI. The imports run on YARN and are scheduled by Oozie. +#### Relational databases -Learn more about it on the [ingesting data from traditional databases](http://gethue.com/importing-data-from-traditional-databases-into-hdfshive-in-just-a-few-clicks/) post. +You can import data from relational databases into an HDFS file or a Hive table with Apache Sqoop. The imports run on YARN and are scheduled by Oozie. This option requires `enable_sqoop=true` in the `[indexer]` section of `hue.ini`. -#### Apache Solr +Learn more in the post on [ingesting data from traditional databases](http://gethue.com/importing-data-from-traditional-databases-into-hdfshive-in-just-a-few-clicks/). -In the past, indexing data into Solr to then explore it with a [Dynamic Dashboard](/user/querying/#dashboards) has been quite difficult. The task involved writing a Solr schema and a Morphlines file then submitting a job to YARN to do the indexing. Often times getting this correct for non trivial imports could take a few days of work. Now with Hue's new feature you can start your YARN indexing job in minutes. +#### Apache Solr indexes -First you’ll need to have a running Solr cluster that Hue is configured with. +You can index data into Solr and then explore it with a [Dynamic Dashboard](/user/querying/#dashboard). The indexer builds the Solr schema and the Morphlines file for you and runs the indexing job on YARN. -Next you’ll need to install these required libraries. To do so place them in a directory somewhere on HDFS and set the path for config_indexer_libs_path under indexer in the Hue ini to match by default, the config_indexer_libs_path value is set to /tmp/smart_indexer_lib. Additionally under indexer in the Hue ini you’ll need to set enable_new_indexer to true. +Before you start, your administrator needs to: - [indexer] +- Configure Hue with a running Solr cluster. +- Place the required libraries in a directory on HDFS and set `config_indexer_libs_path` in the `[indexer]` section of `hue.ini` to that path (the default is `/tmp/smart_indexer_lib`). +- Keep `enable_new_indexer` set to `true` in the `[indexer]` section (this is the default). - # Flag to turn on the morphline based Solr indexer. - enable_new_indexer=false +```ini +[indexer] +# Flag to turn on the morphline based Solr indexer. +## enable_new_indexer=true - # Oozie workspace template for indexing. - ## config_indexer_libs_path=/tmp/smart_indexer_lib +# Filesystem directory containing Solr Morphline indexing libs. +## config_indexer_libs_path=/tmp/smart_indexer_lib +``` -We’ll pick a name for our new collection and select our reviews data file from HDFS. Then we’ll click next. +In this example, you pick a name for the new collection, select the reviews data file from HDFS and click **Next**. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-wizard.png) -Field selection and ETL +**Field selection and ETL** -On this tab we can see all the fields the indexer has picked up from the file. Note that Hue has also made an educated guess on the field type. Generally, Hue does a good job inferring data type. However, we should do a quick check to confirm that the field types look correct. +This tab lists all the fields that the indexer found in the file. Hue also guesses the field types. Check that the types are correct. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-wizard-fields.png) -For our data we’re going to perform 4 operations to make a very searchable Solr Collection. +In this example, you apply four operations to make the collection easy to search. -Convert Date +**Convert date** -This operation is implicit. By setting the field type to date we inform Hue that we want to convert this date to a Solr Date. Hue can convert most standard date formats automatically. If we had a unique date format we would have to define it for Hue by explicitly using the convert date operation. +This operation is implicit. When you set the field type to date, Hue converts the value to a Solr date. Hue converts most standard date formats automatically. For a custom date format, add the convert date operation explicitly. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-date.png) -Translate star ratings to integer ratings +**Translate star ratings to integer ratings** -Under the rating field we’ll change the field type from string to long and click add operation. We’ll then select the translate operation and setup the following translation mapping. +Under the rating field, change the field type from string to long and click add operation. Select the translate operation and set up the following translation mapping. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-translate-date.png) -Grok the city from the full address field +**Grok the city from the full address field** -We’ll add a grok operation to the full address field, fill in the following regex .* (?\w+),.* and set the number of expected fields to +Add a grok operation to the full address field, enter the regex `.* (?\w+),.*` and set the number of expected fields to 1. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-grok.png) -1. In the new child field we’ll set the name to city. This new field will now contain the value matching the city capture group in the regex. +In the new child field, set the name to `city`. This field contains the value of the `city` capture group. -Use a split operation to separate the latitude/longitude field into two separate floating point fields. -Here we have an annoyance. Our data file contains the latitude and longitude of the place that’s being reviewed – Awesome! For some reason though they’ve been clumped into one field with a comma between the two numbers. We’ll use a split operation to grab each individually. Set the split value to ‘,’ and the number of output fields to 2. Then change the child fields’ types to doubles and give them logical names. In this case there’s not really much sense in keeping the parent field so let’s uncheck the “keep in index” box. +**Split the latitude and longitude into two fields** + +The data file has the latitude and longitude in one field, separated by a comma. Add a split operation, set the split value to `,` and the number of output fields to 2. Change the child field types to double and give them clear names. Clear the **keep in index** box for the parent field. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-split.png) -Here we’ll add a geo ip operation and select iso_code as our output. This will give us the country code. +**Add the country from the IP address** -![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-geoip.png) +Add a geo ip operation and select `iso_code` as the output. This gives you the country code. -Before we index, let’s make sure everything looks good with a quick scan of the preview. This can be handy to avoid any silly typos or anything like that. +![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-geoip.png) -Now that we’ve defined our ETL Hue can do the rest. Click index and wait for Hue to index our data. At the bottom of this screen we can see a progress bar of the process. Yellow means our data is currently being indexed and green means it’s done. Feel free to close this window. The indexing will continue on your cluster. +Check the preview to catch typos. Then click **Index** and wait. A progress bar at the bottom of the screen shows the status: yellow means Hue is indexing the data and green means it is done. You can close the window. The indexing continues on the cluster. -Once our data has been indexed into a Solr Collection we have access to all of Hue’s search features and can make a nice analytics dashboard like this one for our data. +After the data is indexed into a Solr collection, you can use all the search features of Hue and build an analytics dashboard. ![Solr Dashboard](https://cdn.gethue.com/uploads/2016/08/indexer-dash.png) **Dependencies** -The indexer libs path is where all required libraries for indexing should be. If you’d prefer you can assemble this directory yourself. There are three main components to the libs directory: +The indexer libs path contains all the libraries required for indexing. You can assemble this directory yourself. It has three main components: 1. JAR files required by the [MapReduceIndexerTool](http://www.cloudera.com/documentation/enterprise/5-5-x/topics/search_mapreduceindexertool.html) -All required jar files should have shipped with CDH. Currently the list of required JARs is: +The required JAR files ship with the Hadoop distribution that provides Solr search. The list of required JARs is: argparse4j-0.4.3.jar readme.txt @@ -189,7 +207,7 @@ All required jar files should have shipped with CDH. Currently the list of requi solr-solrj-4.10.3-cdh5.8.0.jar noggit-0.5.jar -Should this change and you get a missing class error, you can find whatever jar may be missing by grepping all the jars packaged with CDH for the missing class. +If you get a missing class error, search the JARs of your distribution for the missing class. 2. Maxmind GeoLite2 database @@ -201,35 +219,43 @@ Any grok commands can be defined in text files within the grok_dictionaries sub ### Streams -Kafka topics, Streams, Tables can be listed via the [`ksql` connector](/administrator/configuration/connectors/#ksql). +You can list Kafka topics, streams and tables with the [`ksql` connector](/administrator/configuration/connectors/#ksqldb). ### Permissions -Sentry roles and privileges can directly be edited in the Security interface. +You can edit Sentry roles and privileges in the Security interface. -**Note** Apache Sentry is going to be replaced by Apache Ranger in [HUE-8748](https://issues.cloudera.org/browse/HUE-8748). +> **Note:** If the administrator sets `enable_new_storage_browser=true`, the Storage Browser replaces the separate file browser entries in the menu. -#### Sentry Tables +> **Note:** Apache Sentry is being replaced by Apache Ranger. -It can be tricky to grant a new user proper permissions on a secure cluster, let’s walk through the steps to enable any new user for table creation on a kerberized cluster. Depends on your cluster size, creating user and group on each node can be tedious. Here we use pssh (Parallel ssh) for this task. +#### Sentry tables -1. Install the tool and prepare a file which contains all your hosts. +It can be tricky to grant a new user the right permissions on a secure cluster. This walkthrough enables a new user to create tables on a Kerberized cluster. Creating the user and group on each node can be tedious on a large cluster, so the example uses pssh (parallel ssh). -For Mac user: +1. Install the tool and prepare a file that contains all your hosts. + On macOS: + + ```bash brew install pssh + ``` -For Debian or Ubuntu user: + On Debian or Ubuntu: + ```bash sudo apt-get install pssh + ``` + ```bash cat ~/Documents/nodeshue.txt hue-1.test.cloudera.com hue-2.test.cloudera.com hue-3.test.cloudera.com hue-4.test.cloudera.com + ``` -2. Run follow commands to create user: t1 and group: grp1 on your cluster: +2. Run the following commands to create user: t1 and group: grp1 on your cluster: ``` pssh -h ~/Documents/nodeshue.txt -i useradd t1 @@ -251,9 +277,9 @@ For Debian or Ubuntu user: [4] 13:59:28 [SUCCESS] hue-4.test.cloudera.com ``` -3. Create same Hue user: t1 and group: grp1 and make “t1″a member of “grp1”. +3. Create the same Hue user `t1` and group `grp1`, and make `t1` a member of `grp1`. -4. Then log in as any user with sentry admin permission to run following queries in hive editor: +4. Log in as a user with Sentry admin permission and run the following statements in the Hive editor: ``` create role write_role; @@ -261,14 +287,13 @@ For Debian or Ubuntu user: GRANT ALL ON SERVER server1 TO ROLE write_role; ``` -Now “t1” user or any user in “grp1” can log in and create table by running any hive/impala DDL queries or through Hue importer. +Now the user `t1`, or any user in `grp1`, can log in and create tables with Hive or Impala DDL queries or with the importer. +In most cases, you grant narrower permissions than `ALL` on `SERVER`. The next examples create a `read_only_role` and a `read_write_role` for specific databases. -But mostly we would like to grant proper permissions for users instead of `ALL` on `server`. let’s walk through two other examples like `read_only_role` and `read_write_role` for specific databases. +Create the user `t2` in group `grp2` and the user `t3` in group `grp3` on the cluster and in Hue, with similar commands. Then grant the permissions to each group: -Using similar commands to create t2 user in group grp2 and t3 user in group grp3 on cluster and Hue. Then use following statements to grant proper permission to each group: - -1. Read write access to database: ‘s3db’ for any user in group ‘grp3’: +1. Read and write access to the database `s3db` for any user in group `grp3`: ``` create role read_write_s3db_role; @@ -277,7 +302,7 @@ Using similar commands to create t2 user in group grp2 and t3 user in group grp3 GRANT ALL ON URI 'hdfs://hue-1.test.cloudera.com:8020/user/t3' to ROLE read_write_s3db_role; ``` -2. Read only permission for database: ‘default’ for any user in group ‘grp2’: +2. Read-only permission for the database `default` for any user in group `grp2`: ``` create role read_only_defaultDb_role; @@ -286,22 +311,22 @@ Using similar commands to create t2 user in group grp2 and t3 user in group grp3 GRANT REFRESH ON DATABASE default TO ROLE read_only_defaultDb_role; ``` -Now ‘t3’ user can read and create new tables in `database:s3db` while ‘t2’ user can read database: default only. +Now `t3` can read and create tables in the database `s3db`, and `t2` can only read the database `default`. -We can grant those permission through Hue security page too, it should ends like following. +You can also grant these permissions in the Hue Security page. The result looks like this: ![Listing of Sentry Table privileges](https://cdn.gethue.com/uploads/2019/04/HueSecurityRoles.png) -Note: You have to grant URI permission to avoid following error during table creation: +> **Note:** You must grant the URI permission. Otherwise, creating a table fails with this error: Error while compiling statement: FAILED: SemanticException No valid privileges User t3 does not have privileges for CREATETABLE The required privileges: Server=server1->URI=hdfs://hue-1.gce.cloudera.com:8020/user/t3/t3_dir->action=*->grantOption=false; -#### Sentry Solr +#### Sentry Solr privileges -[Apache Solr](http://gethue.com/ui-to-edit-sentry-privilege-of-solr-collections/) privileges can be edited directly via the interface. +You can edit [Apache Solr](http://gethue.com/ui-to-edit-sentry-privilege-of-solr-collections/) privileges in the interface. -For listing collections, query and creating collection: +To list collections, query and create a collection, grant: Admin=*->action=* Collection=*->action=* @@ -325,68 +350,95 @@ End user error when querying a collection without the QUERY privilege End user error when modifying a record without the UPDATE privilege -#### HDFS Acls +#### HDFS ACLs + +You can edit [HDFS ACLs](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#ACLs_.28Access_Control_Lists.29) in the Security app. -Editing [HDFS acls](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#ACLs_.28Access_Control_Lists.29) in the Security app: ![HDFS Acls](https://cdn.gethue.com/docs/user/storage_hdfs_acls.png) ## Data -The File Browser application lets you interact with these file systems HDFS, S3 or ADLS: +The File Browser lets you work with these file systems: HDFS, Amazon S3, Azure (ABFS and ADLS), Google Cloud Storage and Apache Ozone. An administrator must [configure the connection](/administrator/configuration/connectors/#storage) first. You can: -- Create files and directories, upload and download files, upload zip - archives and extract them, rename, move, and delete files and directories. -- Change a file's or directory's owner, group, and - permissions. +- Create files and directories, upload and download files, upload zip archives and extract them, rename, move, copy and delete files and directories. +- Change the owner, group and permissions of a file or directory (HDFS). - View and edit files as text or binary. -- Create external tables or export query results +- Create external tables or export query results. -**Exploring in File Browser** +### Browse files -Once Hue is successfully [configured to connect](/administrator/configuration/connectors/#storage) to the storage, we can view all accessible folders within the account by clicking on the storage root. From here, we can view the existing keys (both directories and files) and create, rename, move, copy, or delete existing directories and files. Additionally, we can directly upload files to the storage. +To browse files: + +1. Open the file browser from the left menu (**Storage Browser** if it is enabled, see below). +2. Click the root of the storage to list the accessible folders. +3. Click a folder to open it, or a file to view it. + +You can then create, rename, move, copy or delete directories and files, and upload files directly to the storage. ![Browse files](https://cdn.gethue.com/uploads/2016/08/image2.png) -**Creating SQL Tables** +### Storage Browser + +When the administrator sets `enable_new_storage_browser=true` in the `[desktop]` section of `hue.ini` (the default is `false`), the left menu shows **Storage Browser** instead of the separate file browser entries. The Storage Browser shows one tab per configured file system. In it you can: + +- Search in the current folder and refresh the list. +- Create a new file or folder. Depending on the location, you can also create a bucket (S3, Google Storage, Ozone), a volume (Ozone) or a file system (ABFS). +- Upload files and download files. +- Rename, copy, move, compress and extract files and folders. +- Preview files and edit text files. +- Change the owner, group and permissions, and set the replication factor (HDFS). +- View the summary of a folder, such as the number of files and the disk space used. +- Move items to the trash, restore them and empty the trash (HDFS). + +> **Note:** The available actions depend on the file system and your permissions. Archive extraction requires `enable_extract_uploaded_archive` to be `true` in the `[filebrowser]` section. It is on by default only when the Oozie app is enabled, and the Storage Browser offers it only for HDFS. -Hue’s table browser import wizard can create external Hive tables directly from files in the storage. This allows the data to be queried via SQL, without moving or copying the data into HDFS or the Hive Warehouse. To create an external table from the storage, navigate to the table browser, select the desired database and then click the plus icon in the upper right. Select a file using the file picker and browse to a file on the storage. +### Create SQL tables from files -Choose your input files’ delimiter and press next. Keep unchecked “Store in Default location” if you want the file to stay intact on the storage, update the column definition options and finally click “Submit” when you’re ready to create the table. Once created, you should see the newly created table details in the table browser. +The table import wizard can create external Hive tables directly from files in the storage. You can then query the data with SQL without copying it into HDFS or the Hive warehouse. + +To create an external table from a file: + +1. Open the Table Browser and select the database. +2. Click the plus icon in the upper right. +3. Use the file picker to select a file on the storage. +4. Choose the delimiter of the file and click **Next**. +5. Clear **Store in Default location** to keep the file where it is. +6. Update the column definitions and click **Submit**. + +The new table appears in the Table Browser. ![Create tables from external files](https://cdn.gethue.com/uploads/2017/11/image4-1.png) -**Saving Query Results** +### Save query results to storage -Now that we have created external Hive tables created from our data, we can jump into either the Hive or Impala editor and start querying the data directly from the storage seamlessly. These queries can join tables. Query results can then easily be saved back to the storage. +After you create external tables, you can query them in the Hive or Impala editor, including joins. You can then save the query results back to the storage. ![Save results to storage](https://cdn.gethue.com/uploads/2017/11/image1-1.png) ### HDFS -Hue is fully compatible with HDFS and is handy for browsing, peeking at file content, upload or downloading data. +You can browse HDFS, peek at the content of files, and upload and download data. ### AWS S3 -Hue can be setup to read and write to a configured S3 account, and users get autocomplete capabilities and can directly query from and save data to S3 without any intermediate moving/copying to HDFS. +When the administrator configures an S3 account, you can read and write data on S3. You get autocomplete, and you can query and save data on S3 without moving it to HDFS first. ### Azure -ADLS v1 as well as ABFS (ADLS v2) are supported. +ABFS (ADLS Gen2) and ADLS Gen1 are supported. -### GCS +### Google Cloud Storage -Google Cloud Storage is currently a work in progress [HUE-8978](https://issues.cloudera.org/browse/HUE-8978) +Google Cloud Storage (`gs://` paths) is supported when the administrator configures it. -### Streams +### Apache Ozone -#### Kafka - -Topics, Streams can be listed via the [`ksql` connector](/administrator/configuration/connectors/#ksql). +Apache Ozone (`ofs://` paths) is supported when the administrator configures it. ### HBase -#### Smart View +#### Smart view The smartview is the view that you land on when you first enter a table. On the left hand side are the row keys and hovering over a row reveals a @@ -433,7 +485,7 @@ operations: ![HBase](https://cdn.gethue.com/uploads/2021/10/hbase_4.png) -#### Smart Searchbar +#### Smart searchbar The "Smart Searchbar" is a sophisticated tool that helps you zero-in on your data. The smart search supports a number of operations. The most @@ -511,7 +563,7 @@ performed exactly like a regular scan, but with a prefixed row. Finally, as a new feature, you can also take full advantage of the -[HBase filtering](denied:about:blank)language, by typing your filter +[HBase filter language](https://hbase.apache.org/book.html#thrift.filter_language), by typing your filter string between curly braces. HBase Browser autocompletes your filters for you so you don't have to look them up every time. You can apply filters to rows or scans. @@ -527,76 +579,68 @@ will suggest next steps to complete your query. ### Indexes -Apache Solr indexes can be created via the [importer](/user/browsing/#data-importer) and are listed in the interface. +You can create Apache Solr indexes with the [importer](/user/browsing/#importing-data). They are listed in the interface. ![Solr Indexer](https://cdn.gethue.com/uploads/2016/08/indexer-op-grok.png) ## Jobs -The Job Browser application lets you to examine multiple types of jobs -jobs running in the cluster. Job Browser presents the job and -tasks in layers for quick access to the logs and troubleshooting. - -### SQL Queries +The Job Browser lets you examine the jobs that run in the cluster. It presents jobs and tasks in layers, so you can quickly reach the logs and troubleshoot. -There are three ways to access the Query browser: +### SQL queries -* Best: Click on the query ID after executing a SQL query in the editor. This will open the mini job browser overlay at the current query. Having the query execution information side by side the SQL editor is especially helpful to understand the performance characteristics of your queries. -* Open the mini job browser overlay and navigate to the queries tab. -* Open the job browser and navigate to the queries tab. - -![Pretty Query Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png) +You can open the query browser in three ways: -There are three ways to access the new browser: +* Click the query ID after you run a SQL query in the editor. This opens the mini job browser overlay at the current query. You see the execution details next to the SQL editor, which helps you understand the performance of your query. This is the recommended way. +* Open the mini job browser overlay and go to the **Queries** tab. +* Open the Job Browser and go to the **Queries** tab. -Best: Click on the query ID after executing a SQL query in the editor. This will open the mini job browser overlay at the current query. Having the query execution information side by side the SQL editor is especially helpful to understand the performance characteristics of your queries. -Open the mini job browser overlay and navigate to the queries tab. -Open the job browser and navigate to the queries tab. +The Impala query section requires `enable_query_browser=true` in the `[jobbrowser]` section of `hue.ini` (the default). The Hive query history requires `enable_hive_query_browser=true`. +![Pretty Query Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png) **Query capabilities** -Display the list of currently running queries on the user’s current Impala coordinator and a certain number of completed queries based on your configuration (25 by default). - -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/12/JB.png) +The query browser displays the list of currently running queries on your current Impala coordinator, and a number of completed queries based on your configuration (25 by default). +![Query list](https://cdn.gethue.com/uploads/2017/12/JB.png) -Display the explain plan which outlines logical execution steps. You can verify here that the execution will not proceed in an unexpected way (i.e. wrong join type, join order, projection order). This can happen if the statistics for the table are out of date as shown in the image below by the mention of “cardinality: unavailable”. You can obtain statistics by running: +Display the explain plan, which shows the logical execution steps. Check that the execution does not proceed in an unexpected way, such as a wrong join type, join order or projection order. This can happen when the table statistics are out of date, as the image below shows with "cardinality: unavailable". To update the statistics, run: - COMPUTE STATS +```sql +COMPUTE STATS +``` -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/11/Explain.png) +![Explain plan](https://cdn.gethue.com/uploads/2017/11/Explain.png) -Display the summary report which shows physical timing and memory information of each operation of the explain plan. You can quickly find bottlenecks in the execution of the query which you can resolve by replacing expensive operations, repartitioning, changing file format or moving data. +Display the summary report, which shows the physical timing and memory information of each operation of the explain plan. You can find bottlenecks and fix them by replacing expensive operations, repartitioning, changing the file format or moving data. -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/11/Summary.png) +![Summary report](https://cdn.gethue.com/uploads/2017/11/Summary.png) -Display the query plan which is a condensed version of the summary report in graphical form. +Display the query plan, a condensed graphical version of the summary report. -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/12/Plan.png) +![Query plan](https://cdn.gethue.com/uploads/2017/12/Plan.png) -Display the memory profile which contains information about the memory usage during the execution of the query. You can use this to determine if the memory available to your query is sufficient. +Display the memory profile, which shows the memory used during the execution of the query. Use it to check whether the memory available to your query is sufficient. -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/11/Memory.png) +![Memory profile](https://cdn.gethue.com/uploads/2017/11/Memory.png) -Display the profile which gives you physical execution of the query in great detail. This view is used to analyze data exchange between the various operator and the performance of the IO (disk, network, CPU). You can use this to reorganize the location of your data (on disk, in memory, different partitions or file formats). +Display the profile, which gives the physical execution of the query in great detail. Use it to analyze data exchange between operators and I/O performance (disk, network, CPU). You can use it to reorganize the location of your data (on disk, in memory, different partitions or file formats). -![Pretty Query Profile](https://cdn.gethue.com/uploads/2017/12/Profile.png) - -Manually close an opened query. +![Query profile](https://cdn.gethue.com/uploads/2017/12/Profile.png) +You can also manually close an open query. ### YARN (Spark, MapReduce, Tez) -Any job running on the Resource Manager will be automatically listed. The information will be fetched accordingly if the job got moved to one of the history servers. - +Any job running on the Resource Manager is listed automatically. If the job moved to a history server, Hue fetches the information from there. -### Oozie Workflow / Schedules +### Oozie workflows and schedules -List submitted workflows, schedules and bundles. +You can list submitted workflows, schedules and bundles. ![Oozie jobs](https://cdn.gethue.com/uploads/2016/04/hue-dash-oozie.png) -### Spark / Livy +### Spark and Livy -List [Spark Livy](/user/querying/#spark) sessions and submitted statements. +You can list [Spark Livy](/user/querying/#spark) sessions and submitted statements. diff --git a/docs/docs-site/content/user/concept/_index.md b/docs/docs-site/content/user/concept/_index.md index 8c9d738727c..c56506cb561 100644 --- a/docs/docs-site/content/user/concept/_index.md +++ b/docs/docs-site/content/user/concept/_index.md @@ -7,7 +7,7 @@ weight: 1 ## Interface -The layout simplifies the interface and is a snappy single page app. +Hue is a single-page app with a simple layout. ![image](/images/hue-4-interface-concept.png) @@ -16,17 +16,15 @@ From top to bottom we have: * Quick action (big blue button), a global search and a notification area on the right * A collapsible hamburger menu that offers links to the various apps and a quick way to import data * An extended quick browse on the left -* The main app area, where the fun is ;) -* A right Assistant panel for the current application. It offers a live help and depends on the currently selected application. For example in the Hive Editor, it shows a quick browse for the used tables in your query, suggestions on how to write better queries, SQL language and UDF built-in documentation. +* The main app area, where you work +* A right Assistant panel for the current application. Its content depends on the selected application. For example, in the Editor it shows the tables used in your query, the SQL language reference, and built-in function (UDF) documentation. -Learn more on the [The Hue 4 user interface in detail](http://gethue.com/the-hue-4-user-interface-in-detail/). +Learn more in [The Hue 4 user interface in detail](http://gethue.com/the-hue-4-user-interface-in-detail/). ### Top search -Have you ever struggled to remember table names related to your project? Does it take much too long to find those columns or views? Hue now lets you easily search for any table, view, or column across all databases in the cluster. With the ability to search across tens of thousands of tables, you're able to quickly find the tables that are relevant for your needs for faster data discovery. - -The new search bar is always accessible on the top of screen, and it offers a document search and metadata search too if Hue is configured to access a metadata server. +Use the search bar at the top of the screen to find any table, view, or column across all databases. It also searches your saved documents. Metadata search is available when your administrator has configured a metadata server. ![Top Search](https://cdn.gethue.com/uploads/2016/04/table_search.png) @@ -35,27 +33,27 @@ The new search bar is always accessible on the top of screen, and it offers a do Data where you need it when you need it. -Find your documents, HDFS and S3 files and more in the left assist panel, right-clicking items will show a list of actions, you can also drag-and-drop a file to get the path in your editor and more. +Use the left assist panel to find your documents, databases, tables, and files. Right-click an item to see its actions. Drag and drop a file to insert its path into the editor. ![Left assist Navigation and drop](https://cdn.gethue.com/uploads/2018/05/HDFS_Context_Change_Path_2.gif) ### Right assist -This assistant content depends on the context of the application selected and will display the current tables or available UDFs. +The content of this panel depends on the selected application. In the Editor, it shows the tables used in the query, the language reference, and the available functions (UDFs). ### Sample popup -This popup offers a quick way to see sample of the data and other statistics on databases, tables, and columns. You can open the popup from the SQL Assist or with a right-click on any SQL object (table, column, function…). +This popup offers a quick way to see sample of the data and other statistics on databases, tables, and columns. Open the popup from the left assist panel or right-click any SQL object (table, column, function) in the editor. ![Sample popup Navigation](https://cdn.gethue.com/uploads/2018/05/SQL_Context_Navigation.gif) ## Documents -Similarly to Google Documents, any document (e.g. SQL Query, Workflow, Dashboard...) opened in the Hue apps can be shared with other users or groups. +Like in Google Docs, you can share any document (for example a SQL query or a dashboard) with other users or groups. ### Sharing -Sharing happens on the main page or via the top right menu of the selected application. +Share a document from the Documents page or from the top-right menu of the application. Two types of sharing permissions exist: @@ -66,123 +64,125 @@ Shared documents will show-up with a little blue icon. ![Sharing](https://cdn.gethue.com/uploads/2019/04/editor_sharing.png) -Along with document sharing, there is another query sharing option via links which is quicker and easier - **Public Links & Gist Sharing** +Two faster options are also available: public links and gist sharing. #### Public Links -Public links are the same as in Google documents which helps in quickly sharing parameterized saved reports, customer analyses links. They only require the recipient to have a Hue login. People can then reuse the queries on their own by executing them to see the results, fork and adapt them. +Public links work like links in Google Docs. Use them to quickly share saved, parameterized reports. The recipient only needs a Hue login. Recipients can run the query to see the results, then fork and adapt it. - No need to select groups or list of individual users -- Read, write permissions +- Read or write permissions - Linked documents won't show up in your home -- Can be turned off globally +- Can be turned off globally by your administrator (`enable_link_sharing`) - Can be combined with traditional user/group sharing -Here is the sharing popup with the public link option on the top when globally enabled: +The sharing dialog shows the public link option at the top when it is enabled: ![Public Link](https://cdn.gethue.com/uploads/2020/03/editor_sharing_popup.png) #### Gist Sharing - SQL Snippet -Answering a question via a query result? Showing some weird data to a slack channel? Gist are a great quick way to quickly share SQL snippets, with a direct link to the SQL editor. +Use a gist to quickly share a SQL snippet, for example in a Slack channel. The gist is a direct link to the Editor. - Works with an SQL snippet: one or more statements -- The link automatically point to the editor and the SQL content -- The query is displayed in the friendlier [presentation mode](/user/querying/#presentation) -- Slack unfurling will show a mini preview (can be turned off globally) -- Gists are stored in a Gist directory in your home +- The link points to the Editor with the SQL content +- The query is displayed in the [presentation mode](/user/querying/#presentation) +- Slack shows a mini preview of the link (your administrator can turn this off with `enable_gist_preview`) +- Gists are stored in a `Gist` directory in your home directory -Here is an example: +To share a gist: -Select a portion of statements to quick share with a Slack channel: +1. Select the statements to share. +2. Open the editor **More** menu and select **Shareable link**. ![Get Sharable Link](https://cdn.gethue.com/uploads/2020/03/editor_sharing_gist_menu.png) -The link to the SQL fragment is automatically generated: +Hue generates the link to the SQL fragment: ![Gist](https://cdn.gethue.com/uploads/2020/03/editor_sharing_gist_popup.png) -Just paste the link in the Slack channel and users will get a mini preview: +Paste the link in a Slack channel to show a mini preview: ![Gist in Slack](https://cdn.gethue.com/uploads/2020/03/editor_gist_slack.png) -Clicking on the link will open-up the SQL selection: +Clicking the link opens the SQL selection: ![Click Gist](https://cdn.gethue.com/uploads/2020/03/editor_gist_open_presentation_mode.png) ### Import / Export -Via the Home page, saved documents can be exported for backups or transferring to another Hue. +Use the Documents page to export saved documents for backup or to transfer them to another Hue. ## Slack -Currently in **Beta**. -This integration with Hue helps users by assisting them with their SQL queries and have better collaboration/discussion with other users via Slack. +**Note**: The Slack integration is in beta. + +This integration helps you share and discuss SQL queries with other users in Slack. -To set-up the Hue App, the **Slack workspace admin** needs to follow these few [steps](/administrator/configuration/server/#improved-slack-app-installation) described in the Admin section. +To set up the Hue app, your Slack workspace administrator must follow the [installation steps](/administrator/configuration/server/#easier-slack-app-installation). -After successfully plugging-in the app with Hue, type `hello hue` or `@Hue help` in the Slack channel to get the list of things the app can assist you with. +After the app is connected to Hue, type `hello hue` or `@Hue help` in the Slack channel to list what the app can do. ![SQL Assistant Help GIF](https://cdn.gethue.com/uploads/2021/06/hello_hue.gif) ### Sharing Query/Gist links -Users can share query links or gists in the Slack channels where the app is also present which then unfurls giving a rich preview of details about the query for other users to collaborate/discuss with. App also provides the result file for the query in the message thread if the result is available. +You can share query links or gists in Slack channels where the Hue app is present. Slack shows a rich preview of the query. If results are available, the app also posts the result file in the message thread. -Open Hue Editor, run some query and copy its link: +Open the Editor, run a query, and copy its link: ![Run Query in Hue](https://cdn.gethue.com/uploads/2021/04/run_query_in_hue.png) -Paste it in the Slack channel for others to get a rich link preview: +Paste the link in the Slack channel to show a rich preview: ![Query Link Preview](https://cdn.gethue.com/uploads/2021/04/query_link_preview.png) -Slack currently does not support markdown tables and potential improvements with inline preview will come when Hue supports result caching via [query tasks](/administrator/administration/reference/#task-server). - -After evaluating a lot of possible fixes (like uploading result image, truncating columns which did not look good, pivoting result table, [uploading result file](https://github.com/slackapi/python-slack-sdk/issues/991) etc.) and seeing their tradeoffs, we chose to have a few sample rows but keep all the columns by pivoting the result table and to compensate for the loss of rows, Hue app gives the result file in the message thread. +Slack does not support Markdown tables. The preview therefore shows a few sample rows with all columns, pivoted. The app posts the full result file in the message thread. ![Message Thread with Result File](https://cdn.gethue.com/uploads/2021/04/message_thread_with_result_file.png) -Users can also share SQL gists links: +You can also share SQL gist links: ![Gist Link](https://cdn.gethue.com/uploads/2021/04/gist_link.png) ![Gist Link Preview](https://cdn.gethue.com/uploads/2021/04/gist_link_preview.png) ### Directly Share from Editor -To reduce the context-switching even more between Slack and Hue for discussions, users can now directly share their desired queries from the Hue editor in the Slack channels. -App sends the gist link in the selected channel and leverages the already present link unfurling for gist links to give a rich preview. +You can share a query from the Editor directly to a Slack channel. The app posts the gist link in the selected channel, and Slack shows a rich preview. Your administrator can turn this off with `share_from_editor`. -Simply create a gist, select a channel and share directly from the Editor: +Create a gist, select a channel, and share it from the Editor: ![Share from Editor GIF](https://cdn.gethue.com/uploads/2021/06/share_from_editor.gif) ### Assistance #### Detecting SQL SELECT statements -To suggest users to share query links instead of copy/pasting SQL statements for discussions/query doubts, the Hue App detects SELECT statements in the message and gives a quick suggestion to share links along with sending a gist link for the detected SQL which then unfurls giving a rich preview. - -In future release, Hue app can improve with better detection and assist more efficiently with detected SQL. For e.g. using parsers to check and extract tables and suggest query optimizations like LIMITS or tables/joins info. +When a message contains a SELECT statement, the Hue app suggests sharing a link instead of pasted SQL. It also posts a gist link for the detected SQL, which Slack shows as a rich preview. ![Detect SQL GIF](https://cdn.gethue.com/uploads/2021/06/detect_sql_select.gif) #### Query Bank -Type `@Hue queries` in the channel to explore the list of important and most used queries from the latest query bank. This query bank is helpful for sharing complex queries with other users to access. +Type `@Hue queries` in the channel to list important and frequently used queries from the latest query bank. ![Query Bank GIF](https://cdn.gethue.com/uploads/2021/06/hue_query_bank.gif) ### Security -Keeping in mind the security aspect, Slack users needs to be a Hue user to have permission to access the above features from the App. This mapping is currently done by checking Slack users' **email prefix as Hue username** and their **email host is same in Hue instance domain** with which the App is plugged-in. +Slack users must also be Hue users to use these features. Hue maps a Slack user to a Hue user when the **email prefix matches the Hue username** and the **email host matches the domain of the Hue instance** connected to the app. -For example, some persona **Alice** having a Hue account with username **alice** can have read access from some Slack account only if the email prefix of that slack user is same and Hue username and the email host is same in the Hue domain i.e. **alice@gethue.com Slack user** can only access **Hue user alice** on **demo.gethue.com**. +For example, the Slack user `alice@gethue.com` can only access the Hue user `alice` on `demo.gethue.com`. ## Settings ### Landing page -Any application or editor can be starred next to its name so that it becomes the default editor and the landing page when logging in. +Star an application or editor next to its name to make it the default editor and the landing page when you log in. ### Changing the language -The language is automatically detected from the Browser or OS. English, Spanish, French, German, Korean, Japanese and Chinese are supported. +Hue detects the language from your browser or operating system. English, Spanish, French, German, Korean, Japanese, Portuguese, Brazilian Portuguese, and Simplified Chinese are supported. + +To set the language manually: -The language can be manual set by a user in the "My Profile" page. Please go to My Profile > Step2 Profile and Groups > Language Preference and choose the language you want. +1. Open **My Profile**. +2. Go to **Step 2: Profile and Groups** (**Step 2: Profile** for non-admin users). +3. Select a **Language Preference**. +4. Save your changes. diff --git a/docs/docs-site/content/user/querying/_index.md b/docs/docs-site/content/user/querying/_index.md index 97a04e170ce..6cb77522621 100644 --- a/docs/docs-site/content/user/querying/_index.md +++ b/docs/docs-site/content/user/querying/_index.md @@ -5,129 +5,119 @@ draft: false weight: 2 --- -Hue's goal is to make Databases & Datawarehouses querying easy and productive. +Use Hue to query databases and data warehouses. You can first explore your data sources with the [browsers](/user/browsing/). -Several apps, each one specialized in a certain type of querying are available. Data sources can be explored first via the [browsers](/user/browsing/). +* The Editor runs SQL queries. It provides autocomplete, syntax checking, and query assistance. +* The Notebook mode of the Editor runs small programming snippets. It is hidden by default (`show_notebooks=false` in `[notebook]`). +* Dashboards visualize indexed data. They can also query SQL databases. -* The Editor shines for SQL queries. It comes with an intelligent autocomplete, risk alerts and self service troubleshooting. -* The Editor is also available in Notebook mode for quickly executing light programming snippets. -* Dashboards are focusing on visualizing indexed data but can also query SQL databases. - -The configuration of the connectors is currently done by the [Administrator](/administrator/configuration/connectors/). +Your [administrator](/administrator/configuration/connectors/) configures the connectors. ## Editor ![Editor](https://cdn.gethue.com/uploads/2019/08/hue_4.5.png) -### Running Queries - -SQL query execution is the primary use case of the Editor. See the list of most common [Databases and Datawarehouses](/administrator/configuration/connectors/). +### Running queries -1. The currently selected statement has a **left blue** border. To execute a portion of a query, highlight one or more query - statements. -2. Click **Execute**. The Query Results window appears - - There is a Log caret on the left of the progress bar - - Expand the **Columns** by clicking on the column label will scroll to the column. Names and types can be filtered - - Select the **Chart** icon to plot the results - - To expand a row, click on the row number - - To lock a row, click on the lock icon in the row number column - - Search either by clicking on the magnifier icon on the results tab, or pressing `Ctrl/Cmd + F` - - See more below how to [refine your results](#Refining Results) +Running SQL is the main use of the Editor. See the list of supported [databases and data warehouses](/administrator/configuration/connectors/). -3. If there are **multiple statements** in the query (separated by semi-colons), click Next in the - multi-statement query pane to execute the remaining statements. +1. Type your query. The active statement has a blue marker in the left gutter. To run only part of the editor content, select one or more statements. +2. Click **Execute**, or press `Ctrl + Enter` (`Cmd + Enter` on macOS). The results appear below the editor: + - Click the caret on the left of the progress bar to show the log. + - Click a column label in the **Columns** list to scroll to that column. You can filter the list by name and type. + - Click the **Chart** icon to plot the results. + - Click a row number to expand the row. + - Click the lock icon in the row number column to lock the row. + - Click the magnifier icon on the results tab, or press `Ctrl/Cmd + F`, to search the results. +3. If the editor contains **multiple statements** separated by semicolons, click **Next** in the multi-statement pane to run the next statement. -When you have multiple statements it's enough to put the cursor in the statement you want to execute, the active statement is indicated with a blue gutter marking. +To run a single statement, put the cursor in it and execute. -**Note**: Use `CTRL/Cmd + ENTER` to execute queries. +For more on working with results, see [Refining](#refining). -**Note**: On top of the logs panel, there is a link to open the query profile in the [Query Browser](/user/browsing/#sql-queries). +> **Note**: A link at the top of the log panel opens the query profile in the [Query Browser](/user/browsing/#sql-queries). ### Results #### Refining -Lock some rows: this will help you compare data with other rows. When you hover a row id, you get a new lock icon. If you click on it, the row automatically sticks to the top of the table. +Lock rows to compare them with other rows. Hover a row number and click the lock icon. The row sticks to the top of the table. ![Result row locking](https://cdn.gethue.com/uploads/2016/08/lock_rows.gif) -The column list follows the result grid, can be filtered by data type and can be resized. +The column list follows the result grid. You can filter it by data type and resize it. ![Smart headers](https://cdn.gethue.com/uploads/2016/08/column_list.gif) -The headers of fields with really long content will follow your scroll position and always be visible. +Headers of columns with long content stay visible as you scroll. ![Cell content search](https://cdn.gethue.com/uploads/2016/08/headers.gif) -You can now search for certain cell values in the table and the results are highlighted. +Search for cell values in the table. Matches are highlighted. -You can activate the new search either by clicking on the magnifier icon on the results tab, or pressing `Ctrl/Cmd + F`. +To search, click the magnifier icon on the results tab, or press `Ctrl/Cmd + F`. ![Virtual cell rendering](https://cdn.gethue.com/uploads/2016/08/search.gif) -The virtual renderer display just the cells you need at that moment. - -The table you see here has hundreds of columns +The grid renders only the visible cells, so tables with hundreds of columns stay responsive. ![Many cells](https://cdn.gethue.com/uploads/2016/08/virtual_renderer.gif) -If the download to Excel or CSV takes too long, you will have a nice message now +If a download to Excel or CSV takes long, Hue shows a message while it prepares the file. ![Many cells](https://cdn.gethue.com/uploads/2016/08/downloadwait.gif) #### Downloading -There are several ways you can export results of a query. +Use the **Export results** (download) icon to get the results of a query. -The most common: +For small results: -* Download to your computer as a CSV or XLS -* Copy the currently fetched rows to the clipboard +* Download as CSV or XLS to your computer. Your administrator sets the row and size limits of downloads. +* Copy the displayed rows to the clipboard. -Two of them offer greater scalability: +For larger results, select **Export** to save the result: - * Export to an empty folder on your cluster's file system. - * Export to a table. You can choose an already existing table or a new one. +* To an empty folder on your file system. +* To a table. You can choose an existing table or a new one. ![Downloading and Exporting Results](https://cdn.gethue.com/uploads/2019/04/editor_export_results.png) ### Autocomplete -To make your SQL editing experience, Hue comes with one of the best SQL autocomplete on the planet. The new autocompleter knows all the ins and outs of the Hive and Impala SQL dialects and will suggest keywords, functions, columns, tables, databases, etc. based on the structure of the statement and the position of the cursor. - -The result is improved completion throughout. We now have completion for more than just SELECT statements, it will help you with the other DDL and DML statements too, INSERT, CREATE, ALTER, DROP etc. +The autocompleter understands the structure of the statement and the cursor position. It suggests keywords, functions, columns, tables, and databases. It supports the Hive and Impala dialects, and covers DDL and DML statements such as `INSERT`, `CREATE`, `ALTER`, and `DROP`, not only `SELECT`. ![Autocomplete and context assist](https://cdn.gethue.com/uploads/2017/07/hue_4_assistant_2.gif) **Smart column suggestions** -If multiple tables appear in the FROM clause, including derived and joined tables, it will merge the columns from all the tables and add the proper prefixes where needed. It also knows about your aliases, lateral views and complex types and will include those. It will now automatically backtick any reserved words or exotic column names where needed to prevent any mistakes. +If several tables appear in the `FROM` clause, including derived and joined tables, Hue merges their columns and adds prefixes where needed. It knows your aliases, lateral views, and complex types. It adds backticks around reserved words and unusual column names. **Smart keyword completion** -The autocompleter suggests keywords based on where the cursor is positioned in the statement. Where possible it will even suggest more than one word at a time, like in the case of IF NOT EXISTS, no one likes to type too much right? In the parts where order matters but the keywords are optional, for instance after FROM tbl, it will list the keyword suggestions in the order they are expected with the first expected one on top. So after FROM tbl the WHERE keyword is listed above GROUP BY etc. +Keyword suggestions depend on the cursor position. Hue can suggest several words at once, such as `IF NOT EXISTS`. Where keywords are optional but ordered, it lists them in the expected order. For example, after `FROM tbl`, `WHERE` is listed above `GROUP BY`. **Functions** -The improved autocompleter will now suggest functions, for each function suggestion an additional panel is added in the autocomplete dropdown showing the documentation and the signature of the function. The autocompleter know about the expected types for the arguments and will only suggest the columns or functions that match the argument at the cursor position in the argument list. +The autocompleter suggests functions. Each suggestion shows the function documentation and signature in an extra panel. It knows the expected argument types and only suggests columns or functions that match the argument at the cursor. SQL functions reference **Sub-queries, correlated or not** -When editing subqueries it will only make suggestions within the scope of the subquery. For correlated subqueries the outside tables are also taken into account. +In a subquery, suggestions are limited to the scope of the subquery. For correlated subqueries, the outer tables are included. **Context popup** -Right click on any fragment of the queries (e.g. a table name) to gets all its metadata information. This is a handy shortcut to get more description or check what types of values are contained in the table or columns. +Right-click any part of a query, such as a table name, to see its metadata. Use it to read descriptions and check the values in a table or column. -It’s quite handy to be able to look at column samples while writing a query to see what type of values you can expect. Hue now has the ability to perform some operations on the sample data, you can now view distinct values as well as min and max values. Expect to see more operations in coming releases. +The popup also shows column samples. You can view distinct, minimum, and maximum values of the sample. ![Sample column popup](https://cdn.gethue.com/uploads/2018/10/sample_context_operations.gif) **Syntax checker** -A little red underline will display the incorrect syntax so that the query can be fixed before submitting. A right click offers suggestions. +Hue underlines incorrect syntax in red so you can fix the query before you run it. Right-click the underlined text to see suggestions. ![Syntax checker](https://cdn.gethue.com/uploads/2018/01/syntax_checkerhigh.png) @@ -135,242 +125,271 @@ A little red underline will display the incorrect syntax so that the query can b **Advanced Settings** -The live autocompletion is fine-tuned for a better experience advanced settings an be accessed via `CTRL +` , (or on Mac `CMD + ,`) or clicking on the '?' icon. +To change autocomplete, syntax checker, dark theme, and other editor settings, press `Ctrl + ,` (`Cmd + ,` on macOS) while the editor has focus. Press `Ctrl + Space` to trigger autocomplete when live autocompletion is off. -The autocompleter talks to the backend to get data for tables and databases etc and caches it to keep it quick. Clicking on the refresh icon in the left assist will clear the cache. This can be useful if a new table was created outside of Hue and is not yet showing-up (Hue will regularly clear his cache to automatically pick-up metadata changes done outside of Hue). +The autocompleter caches table and database metadata from the backend. If a table was created outside of Hue and does not show up, click the refresh icon in the left assist panel to clear the cache. Hue also clears the cache regularly. ### Sharing -Any query can be shared with permissions, as detailed in the [concepts](/user/concept/). +You can share any query with other users and groups. See [Sharing](/user/concept/#sharing). ### Assist - -The Datawarehouse ecosystem is getting more complete with the introduction of transactions. In practice, this means your tables can now support `Primary Keys`, `INSERTs`, `DELETEs` and `UPDATEs` as well as `Partition Keys`. - -Here is a tutorial demoing how Hue's SQL Editor helps you quickly visualize and use these instructions via its [assists](/user/concept/) and [autocomplete](/user/querying/#autocomplete) components. +The assist panel shows primary keys, foreign keys, partition keys, nested types, and views for your tables. These examples use transactional tables, which support `INSERT`, `DELETE`, and `UPDATE` statements. They show how the [assist panels](/user/concept/) and [autocomplete](#autocomplete) help you write them. ![Assist All Keys](https://cdn.gethue.com/uploads/2019/11/sql_column_pk.png) -#### Primary Keys +#### Primary keys -Primary Keys shows up like Partition Keys with the lock icon: +Primary keys show with a lock icon: ![Assist Primary Keys](https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_pks.png) Here is an example of SQL for using them: - CREATE TABLE customer ( - first_name string, - last_name string, - website string, - PRIMARY KEY (first_name, last_name) DISABLE NOVALIDATE - ); +```sql +CREATE TABLE customer ( + first_name string, + last_name string, + website string, + PRIMARY KEY (first_name, last_name) DISABLE NOVALIDATE +); +``` [Apache Kudu](https://kudu.apache.org/) is supported as well: - CREATE TABLE students ( - id BIGINT, - name STRING, - PRIMARY KEY(id) - ) - PARTITION BY HASH PARTITIONS 16 - STORED AS KUDU - TBLPROPERTIES ('kudu.num_tablet_replicas' = '1') - ; +```sql +CREATE TABLE students ( + id BIGINT, + name STRING, + PRIMARY KEY(id) +) +PARTITION BY HASH PARTITIONS 16 +STORED AS KUDU +TBLPROPERTIES ('kudu.num_tablet_replicas' = '1') +; +``` -#### Foreign Keys +#### Foreign keys -When a column value points to another column in another table. e.g. The head of the business unit must exist in the person table: +A foreign key column points to a column in another table. In this example, the head of a business unit must exist in the `person` table: ![Assist Foreign Keys](https://cdn.gethue.com/uploads/2020/03/assist_foreign_keys_icons.png) - CREATE TABLE person ( - id INT NOT NULL, - name STRING NOT NULL, - age INT, - creator STRING DEFAULT CURRENT_USER(), - created_date DATE DEFAULT CURRENT_DATE(), +```sql +CREATE TABLE person ( + id INT NOT NULL, + name STRING NOT NULL, + age INT, + creator STRING DEFAULT CURRENT_USER(), + created_date DATE DEFAULT CURRENT_DATE(), - PRIMARY KEY (id) DISABLE NOVALIDATE - ); + PRIMARY KEY (id) DISABLE NOVALIDATE +); - CREATE TABLE business_unit ( - id INT NOT NULL, - head INT NOT NULL, - creator STRING DEFAULT CURRENT_USER(), - created_date DATE DEFAULT CURRENT_DATE(), +CREATE TABLE business_unit ( + id INT NOT NULL, + head INT NOT NULL, + creator STRING DEFAULT CURRENT_USER(), + created_date DATE DEFAULT CURRENT_DATE(), - PRIMARY KEY (id) DISABLE NOVALIDATE, - CONSTRAINT fk FOREIGN KEY (head) REFERENCES person(id) DISABLE NOVALIDATE - ); + PRIMARY KEY (id) DISABLE NOVALIDATE, + CONSTRAINT fk FOREIGN KEY (head) REFERENCES person(id) DISABLE NOVALIDATE +); +``` -#### Partition Keys +#### Partition keys -Partitioning of the data is a key concept for optimizing the querying. Those special columns are also shown with a key icon: +Partitioning of the data is a key concept for optimizing the querying. Partition columns are shown with a key icon: ![Assist Column Partition Keys](https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_keys.png) Here is an example of SQL for using them: - CREATE TABLE web_logs ( - _version_ BIGINT, - app STRING, - bytes SMALLINT, - city STRING, - client_ip STRING, - code TINYINT, - country_code STRING, - country_code3 STRING, - country_name STRING, - device_family STRING, - extension STRING, - latitude FLOAT, - longitude FLOAT, - `METHOD` STRING, - os_family STRING, - os_major STRING, - protocol STRING, - record STRING, - referer STRING, - region_code BIGINT, request STRING, - subapp STRING, - TIME STRING, - url STRING, - user_agent STRING, - user_agent_family STRING, - user_agent_major STRING, - id STRING - ) - PARTITIONED BY ( `date` STRING); - - INSERT INTO web_logs - PARTITION (`date`='2019-11-14') VALUES - (1480895575515725824,'metastore',1041,'Singapore','128.199.234.236',127,'SG','SGP','Singapore','Other',NULL,1.2930999994277954,103.85579681396484,'GET','Other',NULL,'HTTP/1.1',NULL,'-',0,'GET /metastore/table/default/sample_07 HTTP/1.1','table','2014-05-04T06:35:49Z','/metastore/table/default/sample_07','Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)','Other',NULL,'8836e6ce-9a21-449f-a372-9e57641389b3') - -#### Nested Types - -Complex or Nested Types are handy for storing associated data close together. The assist lets you expand the tree of columns: +```sql +CREATE TABLE web_logs ( + _version_ BIGINT, + app STRING, + bytes SMALLINT, + city STRING, + client_ip STRING, + code TINYINT, + country_code STRING, + country_code3 STRING, + country_name STRING, + device_family STRING, + extension STRING, + latitude FLOAT, + longitude FLOAT, + `METHOD` STRING, + os_family STRING, + os_major STRING, + protocol STRING, + record STRING, + referer STRING, + region_code BIGINT, request STRING, + subapp STRING, + TIME STRING, + url STRING, + user_agent STRING, + user_agent_family STRING, + user_agent_major STRING, + id STRING +) +PARTITIONED BY ( `date` STRING); + +INSERT INTO web_logs +PARTITION (`date`='2019-11-14') VALUES +(1480895575515725824,'metastore',1041,'Singapore','128.199.234.236',127,'SG','SGP','Singapore','Other',NULL,1.2930999994277954,103.85579681396484,'GET','Other',NULL,'HTTP/1.1',NULL,'-',0,'GET /metastore/table/default/sample_07 HTTP/1.1','table','2014-05-04T06:35:49Z','/metastore/table/default/sample_07','Mozilla/5.0 (compatible; phpservermon/3.0.1; +http://www.phpservermonitor.org)','Other',NULL,'8836e6ce-9a21-449f-a372-9e57641389b3') +``` + +#### Nested types + +Complex (nested) types store related data together. Expand the column tree in the assist panel to see their fields: ![Assist Nested Types](https://cdn.gethue.com/uploads/2019/11/sql_columns_assist_nested_types.png) Here is an example of SQL for using them: - CREATE TABLE subscribers ( - id INT, - name STRING, - email_preferences STRUCT>, - addresses MAP>, - orders ARRAY>>> - ) - STORED AS PARQUET +```sql +CREATE TABLE subscribers ( + id INT, + name STRING, + email_preferences STRUCT>, + addresses MAP>, + orders ARRAY>>> +) +STORED AS PARQUET +``` #### Views -It can be sometimes confusing to not recognize that a table is instead a view. Views are shown with this little eye icon: +Views are shown with an eye icon so that you can tell them apart from tables: ![Assist Nested Types](https://cdn.gethue.com/uploads/2019/11/sql_assist_view_icon.png) Here is an example of SQL for using them: - CREATE VIEW web_logs_november AS - SELECT * FROM web_logs - WHERE `date` BETWEEN '2019-11-01' AND '2019-12-01' +```sql +CREATE VIEW web_logs_november AS +SELECT * FROM web_logs +WHERE `date` BETWEEN '2019-11-01' AND '2019-12-01' +``` -#### Transactional Operations +#### Transactional operations -Transactional tables now support these SQL instructions to update the data. +Transactional tables support these statements to change data. ##### Inserts -Here is how to add some data into a table. Previously, he was only possible to do this via LOADING some files. +Add rows to a table: - INSERT INTO TABLE customer - VALUES - ('Elli', 'SQL', 'gethue.com'), - ('John', 'SELECT', 'docs.gethue.com') - ; +```sql +INSERT INTO TABLE customer +VALUES + ('Elli', 'SQL', 'gethue.com'), + ('John', 'SELECT', 'docs.gethue.com') +; +``` ##### Deletes Deletion of rows of data: - DELETE FROM customer - WHERE first_name = 'John'; +```sql +DELETE FROM customer +WHERE first_name = 'John'; +``` ##### Updates -How to update the value of some columns in certain rows: +Update column values in selected rows: - UPDATE customer - SET website = 'helm.gethue.com' - WHERE first_name = 'Elli'; +```sql +UPDATE customer +SET website = 'helm.gethue.com' +WHERE first_name = 'Elli'; +``` -##### Language reference +#### Language reference -You can find the Language Reference in the right assist panel. The right panel itself has a new look with icons on the left hand side and can be minimised by clicking the active icon. +The language reference is in the right assist panel. Click the active icon on the left of the panel to minimize it. -The filter input on top will only filter on the topic titles in this initial version. Below is an example on how to find documentation about joins in select statements. +The filter at the top filters topic titles only. This example finds the documentation about joins in `SELECT` statements: ![Language Reference Panel](https://cdn.gethue.com/uploads/2018/10/impala_lang_ref_joins.gif) -While editing a statement there’s a quicker way to find the language reference for the current statement type, just right-click the first word and the reference appears in a popover below: +To open the reference for the current statement type, right-click the first word of the statement. The reference appears in a popover: ![Language Reference context](https://cdn.gethue.com/uploads/2018/10/impala_lang_ref_context.png) ### Variables -Variables are used to easily configure parameters in a query. They are ideal for saving reports that can be shared or executed repetitively: -**Single Valued** +Use variables to set query parameters at run time. They are ideal for saving reports that you share or run repeatedly. + +**Single value** - select * from web_logs where country_code = "${country_code}" +```sql +select * from web_logs where country_code = "${country_code}" +``` ![Single valued variable](https://cdn.gethue.com/uploads/2017/10/var_defaults.png) -**The variable can have a default value** +**Default value** - select * from web_logs where country_code = "${country_code=US}" +```sql +select * from web_logs where country_code = "${country_code=US}" +``` -**Multi Valued** +**Multiple values** - select * from web_logs where country_code = "${country_code=CA, FR, US}" +```sql +select * from web_logs where country_code = "${country_code=CA, FR, US}" +``` -**In addition, the displayed text for multi valued variables can be changed** +**Custom display text for multiple values** - select * from web_logs where country_code = "${country_code=CA(Canada), FR(France), US(United States)}" +```sql +select * from web_logs where country_code = "${country_code=CA(Canada), FR(France), US(United States)}" +``` ![Multi valued variables](https://cdn.gethue.com/uploads/2018/04/variables_multi.png) -**For values that are not textual, omit the quotes.** +**Non-text values (omit the quotes)** - select * from boolean_table where boolean_column = ${boolean_column} +```sql +select * from boolean_table where boolean_column = ${boolean_column} +``` ### Charting -These visualizations are convenient for plotting chronological data or when subsets of rows have the same attribute: they will be stacked together. +Click the **Chart** icon on the results to plot them. Available chart types: -* Pie -* Bar/Line with pivot -* Timeline -* Scattered plot -* Maps (Marker and Gradient) +* Bar chart +* Timeline chart +* Pie chart +* Scatter plot +* Marker map +* Gradient map ![Charts](https://cdn.gethue.com/uploads/2019/04/editor_charting.png) ### Query troubleshooting -#### Pre-query execution +#### Before running a query **Popular values** -The autocompleter will suggest popular tables, columns, filters, joins, group by, order by etc. based on metadata from Navigator Optimizer. A new “Popular” tab has been added to the autocomplete result dropdown which will be shown when there are popular suggestions available. +> **Note**: Popular values and risk alerts need an optimizer service. The optimizer is off by default (`mode=off` in `[optimizer]`). Ask your administrator whether it is enabled. -This is particularly useful for doing joins on unknown datasets or getting the most interesting columns of tables with hundreds of them. +When an optimizer is enabled, the autocompleter suggests popular tables, columns, filters, joins, `GROUP BY`, and `ORDER BY` clauses. A **Popular** tab appears in the autocomplete dropdown when suggestions are available. + +This helps you join unfamiliar datasets and find the most useful columns in tables with hundreds of columns. ![Popular joins suggestion](https://cdn.gethue.com/uploads/2017/07/hue_4_query_joins.png) ![Popular columns suggestion](https://cdn.gethue.com/uploads/2017/07/hue_4_popular_filter_agg.png) @@ -378,28 +397,30 @@ This is particularly useful for doing joins on unknown datasets or getting the m **Risk alerts** -While editing, Hue will run your queries through Navigator Optimizer in the background to identify potential risks that could affect the performance of your query. If a risk is identified an exclamation mark is shown above the query editor and suggestions on how to improve it is displayed in the lower part of the right assistant panel. +While you type, Hue checks the query in the background for risks that could affect performance. If it finds a risk, an exclamation mark appears above the editor. Suggestions to improve the query appear in the lower part of the right assist panel. ![Query Risk alerts](https://cdn.gethue.com/uploads/2017/07/hue_4_risk_6.gif) -#### During execution +#### While the query runs -The [Query Browser](/user/browsing/#sql-queries) details the plan of the query and the bottle necks. When detected, "Health" risks are listed with suggestions on how to fix them. +The [Query Browser](/user/browsing/#sql-queries) shows the query plan and its bottlenecks. When detected, "Health" risks are listed with suggestions to fix them. ![Pretty Query Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png) #### Tutorial -After finding data in the Catalog and using the Query Assistant, end users might wonder why their queries are taking a lot of time to execute. Build up on top of the Impala profiler, this new feature educates them and surface up more information so that they can be more productive by themselves. Here is a scenario that showcases the flow: +If a query is slow, the Impala query profile helps you find out why. This scenario walks through the profile: **Execution Timeline** -To give you a feel for the new features, we’ll execute a few queries. +Run a few queries: - SELECT * - FROM - transactions1g s07 left JOIN transactions1g s08 - ON ( s07.field_1 = s08.field_1) limit 100 +```sql +SELECT * +FROM + transactions1g s07 left JOIN transactions1g s08 +ON ( s07.field_1 = s08.field_1) limit 100 +``` transactions1g is a 1GB table and the self join with no predicates will force a network transfer of the whole table. @@ -409,7 +430,7 @@ Looking at the profile, you can see a number on the top right of each node that **Detail pane** -On the right hand side, there is now a pane that is closed by default. To open or close press on the header of the pane. There, you will find a list of all the nodes sorted by execution time, which makes it easier to navigate larger execution graphs. This list is clickable and will navigate to the appropriate node. +The detail pane on the right is closed by default. Click its header to open or close it. It lists all nodes sorted by execution time, which helps you navigate large execution graphs. Click a node in the list to jump to it. ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-06-at-4.12.38-PM.png) @@ -429,21 +450,25 @@ The detail pane also contains detailed statistics aggregated per host per node s **Risks** -In the detail pane, for each node, you will find a section titled risks. This section will contain hints on how to improve performance for this operator. Currently, this is not enabled by default. To enable it, go to your Hue ini file and enable this flag: +For each node, the detail pane has a **Risks** section with hints to improve the performance of the operator. It is disabled by default. Your administrator can enable it in the Hue ini file: - [notebook] - enable_query_analysis=true +```ini +[notebook] +enable_query_analysis=true +``` -**CodeGen** +**Codegen** -Let’s look at a few queries and some of the risks that can be identified. +These queries show some of the risks that Hue can identify. - SELECT s07.description, s07.salary, s08.salary, - s08.salary - s07.salary - FROM - sample_07 s07 left outer JOIN sample_08 s08 - ON ( s07.code = s08.code) - where s07.salary > 100000 +```sql +SELECT s07.description, s07.salary, s08.salary, + s08.salary - s07.salary +FROM + sample_07 s07 left outer JOIN sample_08 s08 +ON ( s07.code = s08.code) +where s07.salary > 100000 +``` sample_07 & sample_08 are small sample tables that come with Hue. @@ -457,20 +482,24 @@ Impala compiles SQL requests to native code to execute each node in the graph. O Adding the missing statistics. One way to do this is to execute the following command: - compute stats sample_07; - compute stats sample_08; +```sql +compute stats sample_07; +compute stats sample_08; +``` This is usually the right thing to do, but on larger tables it can be quite expensive. Disable codegen for the query via: - set DISABLE_CODEGEN=true +```sql +set DISABLE_CODEGEN=true +``` ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.52.37-AM.png) After rerunning the query, we see that CodeGen is now gone. -**Join Order** +**Join order** If we open the join node, there’s a warning for wrong join order. @@ -479,16 +508,16 @@ If we open the join node, there’s a warning for wrong join order. Impala prefers having the table with the larger size on the right hand side of the graph, but in this case the reverse is true. Normally, Impala would optimize this automatically, but we saw that the statistics were missing for the tables being joined. There a few ways we could fix this: * Add the missing statistics as described earlier. -* Rewrite the query the change the join order: +* Rewrite the query to change the join order: - ``` - SELECT s07.description, s07.salary, s08.salary, - s08.salary - s07.salary - FROM - sample_08 s08 left outer JOIN sample_07 s07 - ON ( s07.code = s08.code) - where s07.salary > 100000 - ``` +```sql +SELECT s07.description, s07.salary, s08.salary, + s08.salary - s07.salary +FROM + sample_08 s08 left outer JOIN sample_07 s07 +ON ( s07.code = s08.code) +where s07.salary > 100000 +``` ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-9.57.14-AM.png) @@ -498,31 +527,37 @@ The warning is gone and the execution time for the join is down. Impala will execute all of its operators in memory if enough is available. If the execution does not all fit in memory, Impala will use the available disk to store its data temporarily. To see this in action, we’ll use the same query as before, but we’ll set a memory limit to trigger spilling: - set MEM_LIMIT=1g; - select * - FROM - transactions1g s07 left JOIN transactions1g s08 - ON ( s07.field_1 = s08.field_1); +```sql +set MEM_LIMIT=1g; +select * +FROM + transactions1g s07 left JOIN transactions1g s08 +ON ( s07.field_1 = s08.field_1); +``` ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-11.40.24-AM.png) Looking at the join node, we can see that there’s an entry in the risk section about a spilled partition. Typically, the join only has CPU time, but in this case it also has IO time due to the spill. -**Kudu Filtering** +**Kudu filtering** Kudu is one of the supported storage backends for Impala. While Impala stand alone can query a variety of file data formats, Impala on Kudu allows fast updates and inserts on your data, and also is a better choice if small files are involved. When using Impala on Kudu, Impala will push down some of the operations to Kudu to reduce the data transfer between the two. However, Kudu does not support all the operators that Impala support. For example, at the time of writing, Impala support the ‘like’ operator, but Kudu does not. In those cases, all the data that cannot be natively filtered in Kudu is transferred to Impala where it will be filtered. Let’s look at a behavior difference between the two. - SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 - where s07.field_5 LIKE '2000-01%'; +```sql +SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 +where s07.field_5 LIKE '2000-01%'; +``` ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-5.00.59-PM.png) When we look at the graph, we see that on the Kudu node we have both IO, which represent the time spent in Kudu, and CPU, which represent the time spent in Impala, for a total of 2.1s. In the risk section, we can also find a warning that Kudu could not evaluate the predicate. - SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 - where s07.field_5 <= '2000-01-31' and s07.field_5 >= '2000-01-01'; +```sql +SELECT * FROM transactions1g_kudu s07 left JOIN transactions1g_kudu s08 on s07.field_1 = s08.field_1 +where s07.field_5 <= '2000-01-31' and s07.field_5 >= '2000-01-01'; +``` ![Impala Profile](https://cdn.gethue.com/uploads/2019/03/Screen-Shot-2019-03-07-at-4.02.33-PM.png) @@ -536,41 +571,45 @@ You might also have queries where the nodes have short execution time, but the t Looking at the global timeline, we see that the planning phase took 3.8s with most of the time in metadata load. When Impala doesn’t have metadata about a table, which can happen after a user executes: - invalidate metadata; +```sql +invalidate metadata; +``` Impala has to refetch the metadata from the metastore. Furthermore, we see that the second most expensive item at 4.1s is first row fetched. This is the time it took the client, Hue in this case, to fetch the results. While both of these events are not things that a user can change, it’s good to see where the time is spent. -#### Post-query execution +#### After the query runs -A new experimental panel when enabled can offer post risk analysis and recommendation on how to tweak the query for better speed. +An experimental panel can offer risk analysis and recommendations to speed up the query after it ran. It is available only when `enable_query_analysis` is enabled. ### Modes #### Presentation -Turns a list of semi-colon separated queries into an interactive presentation by clicking on the 'Dashboard' icon. It is great for doing presentations with a scenario and live results to prove a point or executing reports containting a suite of sequential queries in one click. +Presentation mode turns a list of semicolon-separated queries into an interactive presentation. Click the 'Dashboard' icon, or press `Ctrl + Shift + P` (`Cmd + Shift + P` on macOS), to switch to or from it. Use it for live demos or to run a series of queries in one click. + +Your administrator can turn it off with `enable_presentation` in `[notebook]`. ![Editor Presentation Mode](https://cdn.gethue.com/uploads/2020/02/editor_presentation_mode.png) #### Dark -Initially this mode is limited to the actual editor area and we’re considering extending this to cover all of Hue. +Dark mode applies to the editor area only. ![Editor Dark Mode](https://cdn.gethue.com/uploads/2018/10/editor_dark_mode.png) -To toggle the dark mode you can either press `Ctrl-Alt-T` or `Command-Option-T` on Mac while the editor has focus. Alternatively you can control this through the settings menu which is shown by pressing `Ctrl-`, or `Command-`, on Mac. +To toggle dark mode, press `Ctrl + Alt + T` (`Cmd + Option + T` on macOS) while the editor has focus. You can also change it in the settings menu, which opens with `Ctrl + ,` (`Cmd + ,` on macOS). ## Dashboard -Dashboards provide an interactive way to query indexed data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks. +Dashboards let you query indexed data interactively. You build the analysis with drag and drop and clicks, without programming. The Dashboard link appears automatically when the Search app is available, or when the administrator sets `has_sql_enabled=true` in the `[dashboard]` section. It can also be set with `is_enabled` in the same section. ![Search Full](https://cdn.gethue.com/uploads/2015/08/search-full-mode.png) -Widgets are interconnected together. This is great for exploring new datasets or monitoring without having to type. +Widgets are connected to each other, so you can explore a new dataset or monitor data without typing. ![Analytics dimensions](https://cdn.gethue.com/uploads/2018/08/dashboard_layout_dnd.gif) -The best supported engine is Apache Solr, then support for SQL databases is getting better. To help add more SQL support, feel free to check the [dashboard connector](/developer/development/#connectors) section. +Apache Solr is the best supported engine. SQL databases are supported when `has_sql_enabled` is set in `[dashboard]` (beta). To add SQL support, see the [dashboard connector](/developer/development/#connectors) section. These tutorials showcase the capabilities: @@ -592,12 +631,13 @@ Some facets can be nested: ### Autocomplete -The top bar support faceted and free word text search, with autocompletion. + +The top bar supports faceted and free-text search with autocomplete. ![Search Autocomplete](https://cdn.gethue.com/uploads/2018/01/dashboard_autocomplete.png) -### Marker Map -Points close to each other are grouped together and will expand when zooming-in. A Yelp-like search filtering experience can also be created by checking the box. +### Marker map +Nearby points are grouped and expand when you zoom in. Select the checkbox to filter the records by the visible map area. ![Marker Map](https://cdn.gethue.com/uploads/2015/08/search-marker-map.png) @@ -611,21 +651,29 @@ Links to the original documents can also be inserted. Add to the record a field Any link - {'type': 'link', 'link': 'gethue.com'} +```text +{'type': 'link', 'link': 'gethue.com'} +``` HBase Browser - {'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527'} - {'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1'} - {'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1', 'col': 'c1'} +```text +{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527'} +{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1'} +{'type': 'hbase', 'table': 'document_demo', 'row_key': '20150527', 'fam': 'f1', 'col': 'c1'} +``` File Browser - {'type': 'hdfs', 'path': '/data/hue/file.txt'} +```text +{'type': 'hdfs', 'path': '/data/hue/file.txt'} +``` Table Catalog - {'type': 'hive', 'database': 'default', 'table': 'sample_07'} +```text +{'type': 'hive', 'database': 'default', 'table': 'sample_07'} +``` ![Data Links](https://cdn.gethue.com/uploads/2015/08/search-link-1024x630.png) @@ -635,28 +683,28 @@ Current selected facets and filters, query strings can be saved with a name with ![Rolling time](https://cdn.gethue.com/uploads/2015/08/search-query-def-1024x507.png) -### ‘Fixed’ or ‘rolling’ window +### Fixed or rolling window -Real time indexing can now shine with the rolling window filter and the automatic refresh of the dashboard every N seconds. See it in action in the real time Twitter indexing with Spark streaming post. +Use a rolling window filter and automatic refresh every N seconds to follow data that is indexed in real time. ![Fixed time](https://cdn.gethue.com/uploads/2015/08/search-fixed-time.png) ### 'More like this' -This feature lets you selected fields you would like to use to find similar records. This is a great way to find similar issues, customers, people... with regard to a list of attributes. +Select the fields to use to find similar records, such as similar issues, customers, or people. ![More like this](https://cdn.gethue.com/uploads/2018/01/solr_more_like_this.png) ## Notebook -The goal of Notebooks is to quickly experiment with small programming snippets (with in particular Spark) and do interactive demos. Its goal is to stay lightweight with regards to other notebook or programming systems. +Use Notebooks to experiment with small programming snippets, in particular Spark, and to run interactive demos. Notebooks are lightweight compared to other notebook systems. -The main advantage is to be able to add snippets of different dialects (e.g. PySpark, Hive SQL...) into a single page: +You can add snippets of different dialects (for example PySpark and Hive SQL) to a single page. The Notebook menu is hidden by default (`show_notebooks=false` in `[notebook]`). ![Notebook mode](https://cdn.gethue.com/uploads/2015/10/notebook-october.png) -Any configured language of the Editor will be available as a dialect. Each snippet has a code editor, with autocomplete, syntax highlighting and other feature like shortcut links to HDFS paths and Hive tables. +Every language configured in the Editor is available as a dialect. Each snippet has a code editor with autocomplete, syntax highlighting, and shortcut links to HDFS paths and Hive tables. ![Notebook Screen](https://cdn.gethue.com/uploads/2015/08/notebook.png) @@ -672,9 +720,9 @@ All the spark-submit, spark-shell, pyspark, sparkR properties of jobs & shells c Hue relies on [Livy](https://livy.incubator.apache.org/) for the interactive Scala, Python, SparkSQL and R snippets. -Livy is an open source REST interface for interacting with Apache Spark from anywhere. It got initially developed in the Hue project but got a lot of traction and was moved to its own project on livy.io. +Livy is an open source REST interface for interacting with Apache Spark from anywhere. It started in the Hue project and is now the separate Apache Livy project. -Make sure that the Notebook and interpreters [configured](/administrator/configuration/connectors/#apache-spark). +Your administrator must [configure](/administrator/configuration/connectors/#apache-spark) the Notebook and its interpreters. #### Livy @@ -686,70 +734,74 @@ As the REST server is running, we can communicate with it. We are on the same ma Let’s list our open sessions - curl localhost:8998/sessions +```bash +curl localhost:8998/sessions - {"from":0,"total":0,"sessions":[]} +{"from":0,"total":0,"sessions":[]} +``` -**Note** You can use +> **Note**: Append `| python -m json.tool` to a command to format the output, for example: - | python -m json.tool - -at the end of the command to prettify the output, e.g.: - - curl localhost:8998/sessions/0 | python -m json.tool +```bash +curl localhost:8998/sessions/0 | python -m json.tool +``` There is zero session. We create an interactive PySpark session - curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" localhost:8998/sessions +```bash +curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type: application/json" localhost:8998/sessions - {"id":0,"state":"starting","kind":"pyspark","log":[]} +{"id":0,"state":"starting","kind":"pyspark","log":[]} +``` Sessions ids are incrementing numbers starting from 0. We can then reference the session later by its id. We check the status of the session until its state becomes idle: it means it is ready to be execute snippet of PySpark: - curl localhost:8998/sessions/0 | python -m json.tool +```bash +curl localhost:8998/sessions/0 | python -m json.tool - % Total % Received % Xferd Average Speed Time Time Time Current +% Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed + Dload Upload Total Spent Left Speed - 100 1185 0 1185 0 0 72712 0 --:--:-- --:--:-- --:--:-- 79000 +100 1185 0 1185 0 0 72712 0 --:--:-- --:--:-- --:--:-- 79000 - { +{ - "id": 5, + "id": 5, - "kind": "pyspark", + "kind": "pyspark", - "log": [ + "log": [ - "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.", + "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.", - "15/09/03 17:44:14 INFO ui.SparkUI: Started SparkUI at http://172.21.2.198:4040", + "15/09/03 17:44:14 INFO ui.SparkUI: Started SparkUI at http://172.21.2.198:4040", - "15/09/03 17:44:14 INFO spark.SparkContext: Added JAR file:/home/romain/projects/hue/apps/spark/java-lib/livy-assembly.jar at http://172.21.2.198:33590/jars/livy-assembly.jar with timestamp 1441327454666", + "15/09/03 17:44:14 INFO spark.SparkContext: Added JAR file:/home/romain/projects/hue/apps/spark/java-lib/livy-assembly.jar at http://172.21.2.198:33590/jars/livy-assembly.jar with timestamp 1441327454666", - "15/09/03 17:44:14 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.", + "15/09/03 17:44:14 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.", - "15/09/03 17:44:14 INFO executor.Executor: Starting executor ID driver on host localhost", + "15/09/03 17:44:14 INFO executor.Executor: Starting executor ID driver on host localhost", - "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54584.", + "15/09/03 17:44:14 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54584.", - "15/09/03 17:44:14 INFO netty.NettyBlockTransferService: Server created on 54584", + "15/09/03 17:44:14 INFO netty.NettyBlockTransferService: Server created on 54584", - "15/09/03 17:44:14 INFO storage.BlockManagerMaster: Trying to register BlockManager", + "15/09/03 17:44:14 INFO storage.BlockManagerMaster: Trying to register BlockManager", - "15/09/03 17:44:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:54584 with 530.3 MB RAM, BlockManagerId(driver, localhost, 54584)", + "15/09/03 17:44:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:54584 with 530.3 MB RAM, BlockManagerId(driver, localhost, 54584)", - "15/09/03 17:44:15 INFO storage.BlockManagerMaster: Registered BlockManager" + "15/09/03 17:44:15 INFO storage.BlockManagerMaster: Registered BlockManager" - ], + ], - "state": "idle" + "state": "idle" - } +} +``` ![Livy Architecture sessions](https://cdn.gethue.com/uploads/2015/09/20150818_scalabythebay.024.png) @@ -757,8 +809,10 @@ We check the status of the session until its state becomes idle: it means it is All the properties supported by spark shells like the number of executors, the memory, etc can be changed at session creation. Their format is the same as when typing spark-shell -h - curl -X POST --data '{"kind": "pyspark", "numExecutors": "3", "executorMemory": "2G"}' -H "Content-Type: application/json" localhost:8998/sessions - {"id":0,"state":"starting","kind":"pyspark","numExecutors":"3","executorMemory":"2G","log":[]} +```bash +curl -X POST --data '{"kind": "pyspark", "numExecutors": "3", "executorMemory": "2G"}' -H "Content-Type: application/json" localhost:8998/sessions +{"id":0,"state":"starting","kind":"pyspark","numExecutors":"3","executorMemory":"2G","log":[]} +``` **Executing statements** @@ -766,62 +820,75 @@ In YARN mode, Livy creates a remote Spark Shell in the cluster that can be acces When the session state is idle, it means it is ready to accept statements! Lets compute 1 + 1 - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' +```bash +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' - {"id":0,"state":"running","output":null} +{"id":0,"state":"running","output":null} +``` We check the result of statement 0 when its state is available - curl localhost:8998/sessions/0/statements/0 +```bash +curl localhost:8998/sessions/0/statements/0 - {"id":0,"state":"available","output":{"status":"ok","execution_count":0,"data":{"text/plain":"2"}}} +{"id":0,"state":"available","output":{"status":"ok","execution_count":0,"data":{"text/plain":"2"}}} +``` -**Note** If the statement is taking less than a few milliseconds, Livy returns the response directly in the response of the POST command. +> **Note**: If a statement takes only a few milliseconds, Livy returns the result directly in the response to the `POST` command. Statements are incrementing and all share the same context, so we can have a sequences - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a = 10"}' +```bash +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a = 10"}' - {"id":1,"state":"available","output":{"status":"ok","execution_count":1,"data":{"text/plain":""}}} +{"id":1,"state":"available","output":{"status":"ok","execution_count":1,"data":{"text/plain":""}}} +``` Spanning multiple statements - curl localhost:8998/sessions/5/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a + 1"}' +```bash +curl localhost:8998/sessions/5/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a + 1"}' - {"id":2,"state":"available","output":{"status":"ok","execution_count":2,"data":{"text/plain":"11"}}} +{"id":2,"state":"available","output":{"status":"ok","execution_count":2,"data":{"text/plain":"11"}}} +``` Let’s close the session to free up the cluster. Note that Livy will automatically inactive idle sessions after 1 hour (configurable). - curl localhost:8998/sessions/0 -X DELETE +```bash +curl localhost:8998/sessions/0 -X DELETE - {"msg":"deleted"} +{"msg":"deleted"} +``` -#### Tutorial: Sharing RDDs +#### Tutorial: sharing RDDs This section shows how to share Spark RDDs and contexts. Livy offers remote Spark sessions to users. They usually have one each (or one by Notebook): - # Client 1 - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' - # Client 2 - curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' - # Client 3 - curl localhost:8998/sessions/2/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' - livy_shared_contexts2 +```bash +# Client 1 +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' +# Client 2 +curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' +# Client 3 +curl localhost:8998/sessions/2/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' +``` ![Livy shared context](https://cdn.gethue.com/uploads/2015/10/livy_shared_contexts2.png) -##### ... and so sharing RDDs +##### ...and so sharing RDDs If the users were pointing to the same session, they would interact with the same Spark context. This context would itself manages several RDDs. Users simply need to use the same session id, e.g. 0, and issue commands there: - # Client 1 - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' +```bash +# Client 1 +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}' - # Client 2 - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' +# Client 2 +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' - # Client 3 - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' +# Client 3 +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"..."}' +``` ![Livy multi rdds](https://cdn.gethue.com/uploads/2015/10/livy_multi_rdds2.png) @@ -833,93 +900,109 @@ Now we can even make it more sophisticated while keeping it simple. Imagine we w To make it prettier, we can wrap it in a few lines of Python and call it ShareableRdd. Then users can directly connect to the session and set or retrieve values. - class ShareableRdd(): +```python +class ShareableRdd(): - def __init__(self): - self.data = sc.parallelize([]) + def __init__(self): + self.data = sc.parallelize([]) - def get(self, key): - return self.data.filter(lambda row: row[0] == key).take(1) + def get(self, key): + return self.data.filter(lambda row: row[0] == key).take(1) - def set(self, key, value): - new_key = sc.parallelize([[key, value]]) - self.data = self.data.union(new_key) + def set(self, key, value): + new_key = sc.parallelize([[key, value]]) + self.data = self.data.union(new_key) +``` set() adds a value to the shared RDD, while get() retrieves it. - srdd = ShareableRdd() +```python +srdd = ShareableRdd() - srdd.set('ak', 'Alaska') - srdd.set('ca', 'California') +srdd.set('ak', 'Alaska') +srdd.set('ca', 'California') - srdd.get('ak') +srdd.get('ak') +``` If using the REST Api directly someone can access it with just these commands: - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"srdd.get(\"ak\")"}' - {"id":3,"state":"running","output":null} +```bash +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"srdd.get(\"ak\")"}' +{"id":3,"state":"running","output":null} - curl localhost:8998/sessions/0/statements/3 - {"id":3,"state":"available","output":{"status":"ok","execution_count":3,"data":{"text/plain":"[['ak', 'Alaska']]"}}} +curl localhost:8998/sessions/0/statements/3 +{"id":3,"state":"available","output":{"status":"ok","execution_count":3,"data":{"text/plain":"[['ak', 'Alaska']]"}}} +``` We can even get prettier data back, directly in json format by adding the %json magic keyword: - curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"data = srdd.get(\"ak\")\n%json data"}' - {"id":4,"state":"running","output":null} +```bash +curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"data = srdd.get(\"ak\")\n%json data"}' +{"id":4,"state":"running","output":null} - curl localhost:8998/sessions/0/statements/4 - {"id":4,"state":"available","output":{"status":"ok","execution_count":2,"data":{"application/json":[["ak","Alaska"]]}}} +curl localhost:8998/sessions/0/statements/4 +{"id":4,"state":"available","output":{"status":"ok","execution_count":2,"data":{"application/json":[["ak","Alaska"]]}}} +``` ##### Even from any languages As Livy is providing a simple REST Api, we can quickly implement a little wrapper around it to offer the shared RDD functionality in any languages. Let’s do it with regular Python: - pip install requests - python +```bash +pip install requests +python +``` Then in the Python shell just declare the wrapper: - import requests - import json - - class SharedRdd(): - """ - Perform REST calls to a remote PySpark shell containing a Shared named RDD. - """ - def __init__(self, session_url, name): - self.session_url = session_url - self.name = name - - def get(self, key): - return self._curl('%(rdd)s.get("%(key)s")' % {'rdd': self.name, 'key': key}) - - def set(self, key, value): - return self._curl('%(rdd)s.set("%(key)s", "%(value)s")' % {'rdd': self.name, 'key': key, 'value': value}) - - def _curl(self, code): - statements_url = self.session_url + '/statements' - data = {'code': code} - r = requests.post(statements_url, data=json.dumps(data), headers={'Content-Type': 'application/json'}) - resp = r.json() - statement_id = str(resp['id']) - while resp['state'] == 'running': - r = requests.get(statements_url + '/' + statement_id) - resp = r.json() - return r.json()['data'] +```python +import requests +import json + +class SharedRdd(): + """ + Perform REST calls to a remote PySpark shell containing a Shared named RDD. + """ + def __init__(self, session_url, name): + self.session_url = session_url + self.name = name + + def get(self, key): + return self._curl('%(rdd)s.get("%(key)s")' % {'rdd': self.name, 'key': key}) + + def set(self, key, value): + return self._curl('%(rdd)s.set("%(key)s", "%(value)s")' % {'rdd': self.name, 'key': key, 'value': value}) + + def _curl(self, code): + statements_url = self.session_url + '/statements' + data = {'code': code} + r = requests.post(statements_url, data=json.dumps(data), headers={'Content-Type': 'application/json'}) + resp = r.json() + statement_id = str(resp['id']) + while resp['state'] == 'running': + r = requests.get(statements_url + '/' + statement_id) + resp = r.json() + return r.json()['data'] +``` Instantiate it and make it point to a live session that contains a ShareableRdd: - states = SharedRdd('http://localhost:8998/sessions/0', 'states') +```python +states = SharedRdd('http://localhost:8998/sessions/0', 'states') +``` And just interact with the RDD transparently: - states.get('ak') - states.set('hi', 'Hawaii') +```python +states.get('ak') +states.set('hi', 'Hawaii') +``` ### Others -**Apache Pig** -Type [Apache Pig](https://pig.apache.org/) latin instructions to load/merge data to perform ETL or Analytics. +These snippet types are available only if your administrator configured them as Oozie-based interpreters. + +**Apache Pig**: Type [Apache Pig](https://pig.apache.org/) Latin statements to load and merge data for ETL or analytics. -**Apache Sqoop** -Run an [SQL import](/user/browsing/#relational-databases) from a traditional relational database via an [Apache Sqoop](https://sqoop.apache.org/) command. +**Apache Sqoop**: Run an [SQL import](/user/browsing/#relational-databases) from a relational database with an [Apache Sqoop](https://sqoop.apache.org/) command. diff --git a/docs/user-guide/user-guide.md b/docs/user-guide/user-guide.md index f0dabd9c71d..95a08e5b3ff 100644 --- a/docs/user-guide/user-guide.md +++ b/docs/user-guide/user-guide.md @@ -1,4 +1,3 @@ - @@ -7,1411 +6,20 @@

Doc > Hue User Guide

-
-
- -[TOC] - -
-
- -# Concept -Hue consists in 4 apps in a single page interface that allow the users to perform data -analyses without losing any context. The goal is to promote self service and stay simple like Excel -so that 80% of the user can find, explore and query data and become more data driven. - -**1. Find or import your data** - -Use the left metadata assists to browse your existing data without losing your editor. The top search will look through your saved queries and matching tables, columns and databases. Objects can be tagged for a quick retrieval or assigning a more “humane” name. If the data does not exist yet, just drag & drop it to trigger the Create Table wizard and to import it in just two steps. - -**2. Query your data** - -When you found your data, the Editor's autocomplete is extremely powerful as they support 90-100% of the language syntax and will highlight any syntax or logical error. The right assistant provides quick previews of the datasets, which columns or JOINs are popular and recommendations on how to type optimized queries. After the querying, refine your results before exporting to S3/HDFS/ADLS or downloaded as CSV/Excel. - -** 4 applications** - -* Editor: The goal of Hue's Editor is to make data querying easy and productive. It focuses on SQL but also supports job submissions. It comes with an intelligent autocomplete, search & tagging of data and query assistance. -* Browsers: Hue's Browsers let you easily search, glance and perform actions on data or jobs in Cloud or on premise clusters. -* Dashboard: Dashboards are an interactive way to explore your data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks. -* Scheduler: The application lets you build workflows and then schedule them to run regularly automatically. A monitoring interface shows the progress, logs and allow actions like pausing or stopping jobs. - - -Each app of Hue can be extended to support your own languages or apps as detailed in the [SDK](../sdk/sdk.html). - -## Interface - -The layout simplifies the interface and is now single page app, and this makes things snappier and unifies the apps together. - -![image](images/hue-4-interface-concept.png) - -From top to bottom we have: - -* A completely redesigned top bar, with a quick action (big blue button), a global search and a notification area on the right -* A collapsible hamburger menu that offers links to the various apps and a quick way to import data -* An extended quick browse on the left -* The main app area, where the fun is ;) -* A right Assistant panel for the current application. It's now enabled for the editors, and in case of Hive for instance, it offers you a live help, a quick browse for the used tables in your query, and much more: if your Hue instance is connected to a SQL Optimizer service like Cloudera Navigator Optimizer, it can offer suggestions on your queries! -* Various applications have been grouped into 4 main conceptual areas: - -Learn more on the [The Hue 4 user interface in detail](http://gethue.com/the-hue-4-user-interface-in-detail/). - - -## Top search - -The new search bar is always accessible on the top of screen, and it offers a document search and metadata search too if Hue is configured to access a metadata server like Cloudera Navigator. - -** Embedded Search & Tagging ** - -Have you ever struggled to remember table names related to your project? Does it take much too long to find those columns or views? Hue now lets you easily search for any table, view, or column across all databases in the cluster. With the ability to search across tens of thousands of tables, you're able to quickly find the tables that are relevant for your needs for faster data discovery. - -In addition, you can also now tag objects with names to better categorize them and group them to different projects. These tags are searchable, expediting the exploration process through easier, more intuitive discovery. - -Through an integration with Cloudera Navigator, existing tags and indexed objects show up automatically in Hue, any additional tags you add appear back in Cloudera Navigator, and the familiar Cloudera Navigator search syntax is supported. - -A top search bar now appears. The autocomplete offers a list of facets and prefills the top values. Pressing enter lists the available objects, which can be opened and explored further in the sample popup, the assist or directly into the table browser app. - -** Granular Search ** - -By default, only tables and views are returned. To search for columns, partitions, databases use the ‘type:' filter. - -Example of searches: - -* table:customer → Find the customer table -* table:tax* tags:finance → List all the tables starting with tax and tagged with ‘finance' -* owner:admin type:field usage → List all the fields created by the admin user that matches the usage string -* parentPath:"/default/web_logs" type:FIELD originalName:b* → List all the columns starting with `b` of the table `web_logs` in the database `default`. - -Learn more on the [Search and Tagging](https://blog.cloudera.com/blog/2017/05/new-in-cloudera-enterprise-5-11-hue-data-search-and-tagging/). - -## Left assist - -Data where you need it when you need it - -You can now find your Hue documents, HDFS and S3 files and more in the left assist panel, right-clicking items will show a list of actions, you can also drag-and-drop a file to get the path in your editor and more. - -## Right assist -This assistant content depends on the context of the application selected and will display the current tables or available UDFs. - -## Sample popup - -This popup offers a quick way to see sample of the data and other statistics on databases, tables, and columns. You can open the popup from the SQL Assist or with a right-click on any SQL object (table, column, function…). In this release, it also opens faster and caches the data. - -## Documents - -Similarly to Google Document, queries, workflows... can be saved and shared with other users. - -### Sharing - -Sharing happens on the main page or via the top right menu of the application. Users and groups with Read or Write permissions can be selected. - -### Import / Export - -Via the Home page, saved documents can be exported for backups or transferring to another Hue. - -## Changing the language - -The language is automatically detected from the Browser or OS. English, Spanish, French, German, Korean, Japanese and Chinese are supported. - -The language can be manual set by a user in the "My Profile" page. Please go to My Profile > Step2 Profile and Groups > Language Preference and choose the language you want. - - -# SQL Editors / Notebook -The goal of Hue's Editor is to make data querying easy and productive. - -It focuses on SQL but also supports job submissions. It comes with an intelligent autocomplete, search & tagging of data and query assistance. - -The [custom SQL Editor page](http://gethue.com/custom-sql-query-editors/) also describes the configuration steps. Any editor can be `starred` -next to its name so that it becomes the default editor and the landing page when logging in. - -First, in your hue.ini file, you will need to add the relevant database connection information under the librdbms section: - - [librdbms] - [[databases]] - [[[postgresql]]] - nice_name=PostgreSQL - name=music - engine=postgresql_psycopg2 - port=5432 - user=hue - password=hue - options={} - -Secondly, we need to add a new interpreter to the notebook app. This will allow the new database type to be registered as a snippet-type in the Notebook app. For query editors that use a Django-compatible database, the name in the brackets should match the database configuration name in the librdbms section (e.g. – postgresql). The interface will be set to rdbms. This tells Hue to use the librdbms driver and corresponding connection information to connect to the database. For example, with the above postgresql connection configuration in the librdbms section, we can add a PostgreSQL interpreter with the following notebook configuration: - - [notebook] - [[interpreters]] - [[[postgresql]]] - name=PostgreSQL - interface=rdbms - -## Concepts -### Running Queries - -![image](images/note.jpg) **Note**: To run a query, you must be logged -in to Hue as a user that also has a Unix user account on the remote -server. - -1. To execute a portion of the query, highlight one or more query - statements. -2. Click **Execute**. The Query Results window appears with the results - of your query. - - To view a log of the query execution, toggle the **Log** caret on the - left of the progress bar. You can use the information in this tab - to debug your query. - - To view the columns of the query, expand the **Columns** icon. Clicking - on the column label will scroll to the column. Names and types can be filtered. - - To expand a row, double click on it or click on the row number. - - To lock a row, click on the lock icon in the row number column. - - Search either by clicking on the magnifier icon on the results tab, or pressing Ctrl/Cmd + F - - [See more how to refine your results](http://gethue.com/new-features-in-the-sql-results-grid-in-hive-and-impala/). - -3. If there are multiple statements in the query (separated by semi-colons), click Next in the - Multi-statement query pane to execute the remaining statements. - -When you have multiple statements it's enough to put the cursor in the statement you want to execute, the active statement is indicated with a blue gutter marking. - -![image](images/note.jpg) **Note**: Use CTRL/Cmd + ENTER to execute queries. - -![image](images/note.jpg) **Note**: Under the logs panel, you can view any -MapReduce or [Impala jobs](#impala-queries) that the query generated. - -### Downloading and Exporting Query Results - -To get things started, press the export icon, the bottom last element of the action bar to the top left of the results. There are several ways you can export results of a query. - -![image](images/note.jpg) Two of them offer great scalability: -1. Export to an empty folder on your cluster's file system. This exports the results using multiple files. In the export icon, choose Export and then All. -2. Export to a table. You can choose an already existing table or a new one. In the export icon, choose Export and then Table. - -![image](images/note.jpg) Two of them offer limited scalability: -1. Export to a file on your cluster's file systems. This exports the results to a single file. In the export icon, choose Export and then First XXX. -2. Download to your computer as a CSV or XLS. This exports the results to a single file in comma-separated values or Microsoft Office Excel format. In the export icon, choose Download as CSV or Download as XLS. - - - -### Advanced Query Settings - -The pane to the top of the Editor lets you specify the following -options: - - -
- - - - - -
DATABASEThe database containing the table definitions.
SETTINGSOverride the Hive and Hadoop default settings. To configure a new -setting: - -
    -
  1. Click Add. -
  2. For Key, enter a Hive or Hadoop configuration variable name. -
  3. For Value, enter the value you want to use for the variable. - -For example, to override the directory where structured Hive query logs -are created, you would enter hive.querylog.location for Key, and a -path for Value. -
- -To view the default settings, click the Settings tab at the top of -the page. For information about Hive configuration variables, see: -[http://wiki.apache.org/hadoop/Hive/AdminManual/Configuration](http://wiki.apache.org/hadoop/Hive/AdminManual/Configuration). -For information about Hadoop configuration variables, see: -[http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml](http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml).
FILE RESOURCESMake files locally accessible at query execution time available on the -Hadoop cluster. Hive uses the Hadoop Distributed Cache to distribute the -added files to all machines in the cluster at query execution time. - -
    -
  1. Click Add to configure a new setting. -
  2. From the Type drop-down menu, choose one of the following: -
      -
    • jar - Adds the specified resources to the Java classpath. -
    • archive - Unarchives the specified resources when - distributing them. -
    • file - Adds the specified resources to the distributed - cache. Typically, this might be a transform script (or similar) - to be executed. - -
    • For Path, enter the path to the file or click - ![image](images/browse.png) to browse and select the file. -
- -![image](images/note.jpg) Note: It is not necessary to specify files -used in a transform script if the files are available in the same path -on all machines in the Hadoop cluster.
USER-DEFINED FUNCTIONSSpecify user-defined functions. Click Add to configure a new -setting. Specify the function name in the Name field, and specify -the class name for Classname. - -You *must* specify a JAR file for the user-defined functions in FILE RESOURCES. - -To include a user-defined function in a query, add a $ (dollar sign) -before the function name in the query. For example, if MyTable is a -user-defined function name in the query, you would type: SELECT $MyTable -
PARAMETERIZATIONIndicate that a dialog box should display to enter parameter values when -a query containing the string $parametername is executed. Enabled by -default.
- -### Autocomplete - -To make your SQL editing experience better we've created a new autocompleter for Hue 3.11. The old one had some limitations and was only aware of parts of the statement being edited. The new autocompleter knows all the ins and outs of the Hive and Impala SQL dialects and will suggest keywords, functions, columns, tables, databases, etc. based on the structure of the statement and the position of the cursor. - -The result is improved completion throughout. We now have completion for more than just SELECT statements, it will help you with the other DDL and DML statements too, INSERT, CREATE, ALTER, DROP etc. - -**Smart column suggestions** - -If multiple tables appear in the FROM clause, including derived and joined tables, it will merge the columns from all the tables and add the proper prefixes where needed. It also knows about your aliases, lateral views and complex types and will include those. It will now automatically backtick any reserved words or exotic column names where needed to prevent any mistakes. - - -**Smart keyword completion** - -The new autocompleter suggests keywords based on where the cursor is positioned in the statement. Where possible it will even suggest more than one word at at time, like in the case of IF NOT EXISTS, no one likes to type too much right? In the parts where order matters but the keywords are optional, for instance after FROM tbl, it will list the keyword suggestions in the order they are expected with the first expected one on top. So after FROM tbl the WHERE keyword is listed above GROUP BY etc. - - -**UDFs** - -The improved autocompleter will now suggest functions, for each function suggestion an additional panel is added in the autocomplete dropdown showing the documentation and the signature of the function. The autocompleter know about the expected types for the arguments and will only suggest the columns or functions that match the argument at the cursor position in the argument list. - - -**Sub-queries, correlated or not** - -When editing subqueries it will only make suggestions within the scope of the subquery. For correlated subqueries the outside tables are also taken into account. - -**All about quality** - -We've fine-tuned the live autocompletion for a better experience and we've introduced some options under the editor settings where you can turn off live autocompletion or disable the autocompleter altogether (if you're adventurous). To access these settings open the editor and focus on the code area, press CTRL + , (or on Mac CMD + ,) and the settings will appear. - -The autocompleter talks to the backend to get data for tables and databases etc. by default it will timeout after 5 seconds but once it has been fetched it's cached for the next time around. The timeout can be adjusted in the Hue server configuration. - -We've got an extensive test suite but not every possible statement is covered, if the autocompleter can't interpret a statement it will be silent and no drop-down will appear. If you encounter a case where you think it should suggest something but doesn't or if it gives incorrect suggestions then please let us know. - -Learn more about it in [Autocompleter for Hive and Impala](http://gethue.com/brand-new-autocompleter-for-hive-and-impala/). - -### Variables -Variables are used to easily configure parameters in a query. They can be of two types: - - -Single Valued -
-select * from web_logs where country_code = "${country_code}"
-
-The variable can have a default value. -
-select * from web_logs where country_code = "${country_code=US}"
-
-Multi Valued -
-select * from web_logs where country_code = "${country_code=CA, FR, US}"
-
-In addition, the displayed text for multi valued variables can be changed. -
-select * from web_logs where country_code = "${country_code=CA(Canada), FR(France), US(United States)}"
-
-For values that are not textual, omit the quotes. -
-select * from boolean_table where boolean_column = ${boolean_column}
-
- -### Syntax checker - -A little red underline will display the incorrect syntax so that the query can be fixed before submitting. A right click offers suggestions. - -### Query Assist - -Read more about the [Query Assistant with Navigator Optimizer Integration -](https://blog.cloudera.com/blog/2017/08/new-in-cloudera-enterprise-5-12-hue-4-interface-and-query-assistant/). - -### Charting - -These visualizations are convenient for plotting chronological data or when subsets of rows have the same attribute: they will be stacked together. - -* Pie -* Bar/Line with pivot -* Timeline -* Scattered plot -* Maps (Marker and Gradient) - -Read more about extending [charts](../sdk/sdk.html). - -### Risk Alerts - -The autocompleter will suggest popular tables, columns, filters, joins, group by, order by etc. based on metadata from Navigator Optimizer. A new “Popular” tab has been added to the autocomplete result dropdown which will be shown when there are popular suggestions available. - -**Risk and suggestions** - -While editing, Hue will run your queries through Navigator Optimizer in the background to identify potential risks that could affect the performance of your query. If a risk is identified an exclamation mark is shown above the query editor and suggestions on how to improve it is displayed in the lower part of the right assistant panel. - -### Presentation Mode - -Turns a list of semi-colon separated queries into an interactive presentation. It is great for doing demos or basic reporting. - -## SQL Databases - -Use the query editor with any database. - -### Hive -### Impala -### MySQL -### Oracle -### KSQL / Kafka SQL -### Solr SQL - -With Solr 5+, query collections like we would query a regular Hive or Impala table. - -[Read more about it here](http://gethue.com/sql-editor-for-solr-sql/). - -As Solr SQL is pretty recent, there are some caveats, notably Solr lacks support of: - -* SELECT * -* WHERE close with a LIKE -* resultset pagination - -which prevents a SQL UX experience comparable to the standard other databases (but we track it in [HUE-3686](https://issues.cloudera.org/browse/HUE-3686)). - -### Trino (formerly Presto SQL) - -Trino is a high performance, distributed SQL query engine for big data. - -[Read more about it here](https://trino.io/) - -### PostgreSQL -### Redshift -### BigQuery -### AWS Athena -### Spark SQL -### Phoenix -### Kylin -Apache Kylin is an open-source online analytical processing (OLAP) engine. -See how to configure the [Kylin Query Editor](http://gethue.com/using-hue-to-interact-with-apache-kylin/). - -### Others -Extend with SQL Alchemy, JDBC or build your own [connectors](../sdk/sdk.html#sql). - -## Jobs - -The Editor application enables you to create and submit jobs to -the cluster. You can include variables with your jobs to enable -you and other users to enter values for the variables when they run your -job. - -All job design settings except Name and Description support the use of -variables of the form $variable\_name. When you run the job, a dialog -box will appear to enable you to specify the values of the variables. - - - - - - - - - -
NameIdentifies the job and its collection of properties and parameters.
DescriptionA description of the job. The description is displayed in the dialog box -that appears if you specify variables for the job.
AdvancedAdvanced settings:
  • Is shared- Indicate whether to share the action with all users.
  • Oozie parameters - parameters to pass to Oozie
PrepareSpecifies paths to create or delete before starting the workflow job.
ParamsParameters to pass to a script or command. The parameters are expressed -using the JSP 2.0 Specification (JSP.2.3) Expression -Language, -allowing variables, functions, and complex expressions as parameters.
Job PropertiesJob properties. To set a property value, click Add Property.
  1. Property name - a configuration property name. This field provides autocompletion, so you can type the first few characters of a property name and then select the one you want from the drop-down - list.
  2. Valuethe property value.
FilesFiles to pass to the job. Equivalent to the Hadoop -files option.
ArchivesFiles to pass to the job. Archives to pass to the job. Equivalent to the Hadoop -archives option.
- -### MapReduce - -A MapReduce job design consists of MapReduce functions written in Java. -You can create a MapReduce job design from existing mapper and reducer -classes without having to write a main Java class. You must specify the -mapper and reducer classes as well as other MapReduce properties in the -Job Properties setting. - - - -
Jar pathThe fully-qualified path to a JAR file containing the classes that -implement the Mapper and Reducer functions.
- -### Java - -A Java job design consists of a main class written in Java. - - - - - - -
Jar pathThe fully-qualified path to a JAR file containing the main class.
Main classThe main class to invoke the program.
ArgsThe arguments to pass to the main class.
Java optsThe options to pass to the JVM.
- -### Pig - -A Pig job design consists of a Pig script. - - - -
Script nameScript name or path to the Pig script.
- -### Sqoop - -A Sqoop job design consists of a Sqoop command. - - - -
CommandThe Sqoop command.
- -### Shell - -A Shell job design consists of a shell command. - - - - -
CommandThe shell command.
Capture outputIndicate whether to capture the output of the command.
- -[Read more about it here](http://gethue.com/use-the-shell-action-in-oozie/). - -### DistCp - -A DistCp job design consists of a DistCp command. - - -### Spark - -#### Batch - -This is a quick way to submit any Jar or Python jar/script to a cluster via the Scheduler or Editor. - -How to run Spark jobs with Spark on YARN? This often requires trial and error in order to make it work. - -Hue is leveraging Apache Oozie to submit the jobs. It focuses on the yarn-client mode, as Oozie is already running the spark-summit command in a MapReduce2 task in the cluster. You can read more about the Spark modes here. - -[Here is how to get started successfully](http://gethue.com/how-to-schedule-spark-jobs-with-spark-on-yarn-and-oozie/). -And how to use the [Spark Action](http://gethue.com/use-the-spark-action-in-oozie/). - -#### Interactive - -Hue relies on [Livy](http://livy.io/) for the interactive Scala, Python and R snippets. - -Livy got initially developed in the Hue project but got a lot of traction and was moved to its own project on livy.io. Here is a tutorial on how to use a notebook to perform some Bike Data analysis. - -Read more about it: - -* [How to use the Livy Spark REST Job Server API for doing some interactive Spark with curl](http://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark-2-2/) -* [How to use the Livy Spark REST Job Server API for submitting batch jar, Python and Streaming Jobs](http://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-submitting-batch-jar-python-and-streaming-spark-jobs/) - -Make sure that the Notebook and interpreters are set in the hue.ini, and Livy is up and running: - -
-[spark]
-  # Host address of the Livy Server.
-  livy_server_host=localhost
-
-[notebook]
-
- ## Show the notebook menu or not
- show_notebooks=true
-
-[[interpreters]]
-    # Define the name and how to connect and execute the language.
-
-    [[[hive]]]
-      # The name of the snippet.
-      name=Hive
-      # The backend connection to use to communicate with the server.
-      interface=hiveserver2
-
-   [[[spark]]]
-     name=Scala
-     interface=livy
-
-    [[[pyspark]]]
-      name=PySpark
-      interface=livy
-
- -# Data Importer - -The goal of the importer is to allow ad hoc queries on data not yet in the clusters thereby expedite self-service analytics. - -If you want to import your own data instead of installing the sample -tables, open the importer from the left menu or from the little `+` in the left assist. - -If you've ever struggled with creating new SQL tables from files, you'll be happy to learn that this is now much easier. The wizard has been revamped to two simple steps and also offers more formats. Now users just need to: - -1. Select a source type -2. Select the type of object for the destination - -And that's it! - -To learn more, watch the video on [Data Import Wizard](http://gethue.com/import-data-to-be-queried-via-the-self-service-drag-drop-create-table-wizard/). - -## SQL Tables - -Although you can create tables by executing the appropriate Hive HQL DDL -query commands, it is easier to create a table using the create table wizard. - -**From a File** - -If you've ever struggled with creating new SQL tables from files, you'll be happy to learn that this is now much easier. With the latest Hue release, you can now create these in an ad hoc way and thereby expedite self-service analytics. The wizard has been revamped to two simple steps and also offers more formats. Now users just need to: - -1. In the Importer Manager selects source from a 'File' -1. Select the type of table - -Files can be dragged & dropped, selected from HDFS or S3 (if configured), and their formats are automatically detected. The wizard also assists when performing advanced functionalities like table partitioning, Kudu tables, and nested types. - - -**Manually** - -1. In the Importer Manager selects 'Manually' -2. Follow the instructions in the wizard to create the table. The basic - steps are: - - Name the table. - - Choose the record format. - - Configure record serialization by specifying delimiters for - columns, collections, and map keys. - - Choose the file format. - - Specify the location for your table's data. - - Specify the columns, providing a name and selecting the type for - each column. - - Specify partition columns, providing a name and selecting the - type for each column. - - -## Indexing - -In the past, indexing data into Solr to then explore it with a [Dynamic Dashboard](http://gethue.com/search-dashboards/) has been quite difficult. The task involved writing a Solr schema and a Morphlines file then submitting a job to YARN to do the indexing. Often times getting this correct for non trivial imports could take a few days of work. Now with Hue's new feature you can start your YARN indexing job in minutes. This tutorial offers a step by step guide on how to do it. - -[Read more about it here](http://gethue.com/easy-indexing-of-data-into-solr/). - -## Traditional Databases - -Read more about [ingesting data from traditional databases](http://gethue.com/importing-data-from-traditional-databases-into-hdfshive-in-just-a-few-clicks/). - - -# Dashboards -Dashboards are an interactive way to explore your data quickly and easily. No programming is required and the analysis is done by drag & drops and clicks. - -Read more about [Dashboards](http://gethue.com/search-dashboards/). - -## Concepts - -Simply drag & drop widgets that are interconnected together. This is great for exploring new datasets or monitoring without having to type. - -### Importing - -Any CSV file can be dragged & dropped and ingested into an index in a few clicks via the Data Import Wizard [link]. The indexed data is immediately queryable and its facets/dimensions will be very fast to explore. - -### Browsing - -The Collection browser got polished in the last releases and provide more information on the columns. The left metadata assist of Hue 4 makes it handy to list them and peak at their content via the sample popup. - -### Querying - -The search box support live prefix filtering of field data and comes with a Solr syntax autocomplete in order to make the querying intuitive and quick. Any field can be inspected for its top values of statistic. This analysis happens very fast as the data is indexed. - -## Databases - -### Solr - -#### Autocomplete - -The top search bar offers a [full autocomplete](http://gethue.com/intuitively-discovering-and-exploring-a-wine-dataset-with-the-dynamic-dashboards/) on all the values of the index. - -#### More Like This -The “More like This” feature lets you selected fields you would like to use to find similar records. This is a great way to find similar issues, customers, people... with regard to a list of attributes. - -### SQL - -## Reports - -This is work in progress but dashboards will soon offer a classic reporting option. - -## SDK -Read more about extending [connectors](../sdk/sdk.html#dashboard). - - -# Browsers -Hue's Browsers powers your Data Catalog. They let you easily search, glance and perform actions on data or jobs in Cloud or on premise clusters. - -## Tables - -The Table Browser enables you to manage the databases, -tables, and partitions of the metastore shared by -the Hive and Impala. You can use Metastore -Manager to perform the following operations: - -- Databases - - Select a database - - Create a database - - Drop databases - -- Tables - - Create tables - - Browse tables - - Drop tables - - Browse table data and metadata (columns, partitions...) - - Import data into a table - - [Filter, Sort and Browse Partitions](http://gethue.com/filter-sort-browse-hive-partitions-with-hues-metastore/) - - -## Files - -The File Browser application lets you browse and manipulate files and -directories in the Hadoop Distributed File System (HDFS), S3 or ADLS. -With File Browser, you can: - -- Create files and directories, upload and download files, upload zip - archives, and rename, move, and delete files and directories. You - can also change a file's or directory's owner, group, and - permissions. See [Files and Directories](#filesAndDirectories). -- Search for files, directories, owners, and groups. See [Searching - for Files and Directories](#searching). -- View and edit files as text or binary. See [Viewing and Editing - Files](#viewAndEdit). - -### File systems -#### HDFS -#### S3 - -Hue can be setup to read and write to a configured S3 account, and users get autocomplete capabilities and can directly query from and save data to S3 without any intermediate moving/copying to HDFS. - -[Read more about it](http://gethue.com/introducing-s3-support-in-hue/). - -** Create Hive Tables Directly From S3** -Hue's Metastore Import Data Wizard can create external Hive tables directly from data directories in S3. This allows S3 data to be queried via SQL from Hive or Impala, without moving or copying the data into HDFS or the Hive Warehouse. - -To create an external Hive table from S3, navigate to the Metastore app, select the desired database and then click the “Create a new table from a file” icon in the upper right. - -Enter the table name and optional description, and in the “Input File or Directory” filepicker, select the S3A filesystem and navigate to the parent directory containing the desired data files and click the “Select this folder” button. The “Load Data” dropdown should automatically select the “Create External Table” option which indicates that this table will directly reference an external data directory. - -Choose your input files' delimiter and column definition options and finally click “Create Table” when you're ready to create the Hive table. Once created, you should see the newly created table details in the Metastore. +
-**Save Query Results to S3** +# Hue user guide -Now that we have created external Hive tables created from our S3 data, we can jump into either the Hive or Impala editor and start querying the data directly from S3 seamlessly. These queries can join tables and objects that are backed either by S3, HDFS, or both. Query results can then easily be saved back to S3. +This page is no longer maintained. The Hue user documentation now lives on [docs.gethue.com](https://docs.gethue.com/user/). +The sources of the current documentation are in the [`docs/docs-site/content/user`](https://github.com/cloudera/hue/tree/master/docs/docs-site/content/user) directory of the Hue repository. -** S3 Configuration ** +Start with these topics: -[Hue S3 Documentation](../admin-manual/manual.html#s3). +- [Concept and overview](https://docs.gethue.com/user/) +- [Browsing data and jobs](https://docs.gethue.com/user/browsing/): tables, files, importer and Job Browser +- [Querying data](https://docs.gethue.com/user/querying/): SQL editors, dashboards and scheduling - -#### ADLS - -Learn more about it on the [ADLS integration post](http://gethue.com/browsing-adls-data-querying-it-with-sql-and-exporting-the-results-back-in-hue-4-2/). - - Users gets autocomplete capabilities and more: - -** Exploring ADLS in Hue's file browser ** -Once Hue is successfully configured to connect to ADLS, we can view all accessible folders within the account by clicking on the ADLS root. From here, we can view the existing keys (both directories and files) and create, rename, move, copy, or delete existing directories and files. Additionally, we can directly upload files to ADLS. - -** Create Hive Tables Directly From ADLS ** -Hue's table browser import wizard can create external Hive tables directly from files in ADLS. This allows ADLS data to be queried via SQL from Hive or Impala, without moving or copying the data into HDFS or the Hive Warehouse. To create an external Hive table from ADLS, navigate to the table browser, select the desired database and then click the plus icon in the upper right. Select a file using the file picker and browse to a file on ADLS. - -** Save Query Results to ADLS ** -Now that we have created external Hive tables created from our ADLS data, we can jump into either the Hive or Impala editor and start querying the data directly from ADLS seamlessly. These queries can join tables and objects that are backed either by ADLS, HDFS, or both. Query results can then easily be saved back to ADLS. - - -** ADLS Configuration ** - -[Hue ADLS Documentation](../admin-manual/manual.html#adls). - - -### Files and Directories - -You can use File Browser to view the input and output files of your -MapReduce jobs. Typically, you can save your output files in /tmp or in -your home directory if your system administrator set one up for you. You -must have the proper permissions to manipulate other user's files. - -#### Creating Directories - -1. In the File Browser window, select **New > Directory**. -2. In the **Create Directory** dialog box, enter a directory name and - then click **Submit**. - -#### Changing Directories - -- Click the directory name or parent directory dots in the **File - Browser** window. -- Click the ![image](images/edit.png) icon, type a directory name, and - press **Enter**. - -To change to your home directory, click **Home** in the path field at -the top of the **File Browser** window. - -![image](images/note.jpg) **Note**: - -The **Home** button is disabled if you do not have a home directory. Ask -a Hue administrator to create a home directory for you. - -#### Creating Files - -1. In the File Browser window, select **New > File**. -2. In the **Create File** dialog box, enter a file name and then click - **Submit**. - - -#### Uploading Files - -You can upload text and binary files to the HDFS. - -1. In the **File Browser** window, browse to the directory where you - want to upload the file. -2. Select **Upload \> Files**. -3. In the box that opens, click **Upload a File** to browse to and - select the file(s) you want to upload, and then click **Open**. - -#### Downloading Files - -You can download text and binary files to the HDFS. - -1. In the **File Browser** window, check the checkbox next to the file - you want to download. -2. Click the **Download** button. - -### Uploading Zip Archives - -You can extract zip archives to the HDFS. The archive is -extracted to a directory named archivename. - -1. In the **File Browser** window, browse to the directory where you - want to upload the archive. -2. Select **Upload > Zip file**. -3. In the box that opens, click **Upload a zip file** to browse to and - select the archive you want to upload, and then click **Open**. - -### Trash Folder - -File Browser supports the HDFS trash folder (*home directory*/.Trash) to -contain files and directories before they are permanently deleted. Files -in the folder have the full path of the deleted files (in order to be -able to restore them if needed) and checkpoints. The length of time a -file or directory stays in the trash depends on HDFS properties. - -In the **File Browser** window, click ![image](images/fbtrash.png). - - -### Changing Owner, Group, or Permissions - -![image](images/note.jpg) **Note**: - -Only the Hadoop superuser can change a file's or directory's owner, -group, or permissions. The user who starts Hadoop is the Hadoop -superuser. The Hadoop superuser account is not necessarily the same as a -Hue superuser account. If you create a Hue user (in User Admin) with the -same user name and password as the Hadoop superuser, then that Hue user -can change a file's or directory's owner, group, or permissions. - -**Owner or Group** - -1. In the **File Browser** window, check the checkbox next to the - select the file or directory whose owner or group you want to - change. -2. Choose **Change Owner/Group** from the Options menu. -3. In the **Change Owner/Group** dialog box: - - Choose the new user from the **User** drop-down menu. - - Choose the new group from the **Group** drop-down menu. - - Check the **Recursive** checkbox to propagate the change. - -4. Click **Submit** to make the changes. - -**Permissions** - -1. In the **File Browser** window, check the checkbox next to the file - or directory whose permissions you want to change. -2. Click the **Change Permissions** button. -3. In the **Change Permissions** dialog box, select the permissions you - want to assign and then click **Submit**. - -### Viewing and Editing Files - -You can view and edit files as text or binary. - - -**View** - -1. In the **File Browser** window, click the file you want to view. - File Browser displays the first 4,096 bytes of the file in the - **File Viewer** window. - - If the file is larger than 4,096 bytes, use the Block navigation - buttons (First Block, Previous Block, Next Block, Last Block) to - scroll through the file block by block. The **Viewing Bytes** - fields show the range of bytes you are currently viewing. - - To switch the view from text to binary, click **View as Binary** - to view a hex dump. - - To switch the view from binary to text, click **View as Text**. - -**Edit** - -1. If you are viewing a text file, click **Edit File**. File Browser - displays the contents of the file in the **File Editor** window. -2. Edit the file and then click **Save** or **Save As** to save the - file. - -## Indexes / Collections - -## Sentry Permissions - -Sentry roles and privileges can directly be edited in the Security interface. - -### SQL - -[Hive UI](http://gethue.com/apache-sentry-made-easy-with-the-new-hue-security-app/). - -### Solr - -[Solr](http://gethue.com/ui-to-edit-sentry-privilege-of-solr-collections/) privileges can be edited directly via the interface. - -For listing collections, query and creating collection: - - Admin=*->action=* - Collection=*->action=* - Schema=*->action=* - Config=*->action=* - - -## Jobs - -The Job Browser application lets you to examine multiple types of jobs -jobs running in the cluster. Job Browser presents the job and -tasks in layers. The top layer is a list of jobs, and you can link to a -list of that job's tasks. You can then view a task's attempts and the -properties of each attempt, such as state, start and end time, and -output size. To troubleshoot failed jobs, you can also view the logs of -each attempt. - - -If there are jobs running, then the Job Browser list appears. - -### Dashboard - -- To filter the jobs by their state (such as **Running** or - **Completed**), choose a state from the **Job status** drop-down - menu. -- To filter by a user who ran the jobs, enter the user's name in the - **User Name** query box. -- To filter by job name, enter the name in the **Text** query box. -- To clear the filters, choose **All States** from the **Job status** - drop-down menu and delete any text in the **User Name** and **Text** - query boxes. - -### Viewing Job Information - -![image](images/note.jpg) **Note**: At any level you can view the log -for an object by clicking the ![image](images/log.png) icon in the Logs -column. - -**To view job information for an individual job:** - -1. In the **Job Browser** window, click **View** at the right of the - job you want to view. This shows the **Job** page for the job, with - the recent tasks associated with the job are displayed in the - **Tasks** tab. -2. Click the **Logs** tab to view the logs for this job. -3. Click the **Counters** tab to view the counter metrics for the job. - - -### Types -#### YARN (Spark, MapReduce) -#### Impala Queries - -There are three ways to access the new browser: - -Best: Click on the query ID after executing a SQL query in the editor. This will open the mini job browser overlay at the current query. Having the query execution information side by side the SQL editor is especially helpful to understand the performance characteristics of your queries. -Open the mini job browser overlay and navigate to the queries tab. -Open the job browser and navigate to the queries tab. - -Query capabilities - -* Display the list of currently running queries on the user's current Impala coordinator and a certain number of completed queries based on your configuration (25 by default). -* Display the summary report which shows physical timing and memory information of each operation of the explain plan. You can quickly find bottlenecks in the execution of the query which you can resolve by replacing expensive operations, repartitioning, changing file format or moving data. -* Display the query plan which is a condensed version of the summary report in graphical form -* Display the memory profile which contains information about the memory usage during the execution of the query. You can use this to determine if the memory available to your query is sufficient. -* Display the profile which gives you physical execution of the query in great detail. This view is used to analyze data exchange between the various operator and the performance of the IO (disk, network, CPU). You can use this to reorganize the location of your data (on disk, in memory, different partitions or file formats). -* Manually close an opened query. - -Read more about it on [Browsing Impala Query Execution within the SQL Editor -](http://gethue.com/browsing-impala-query-execution-within-the-sql-editor/). - -#### Workflow / Schedules (Oozie) - -List submitted workflows, schedules and bundles. - -#### Livy / Spark - -List Livy sessions and submitted statements. - -# Scheduler -The application lets you build workflows and then schedule them to run regularly automatically. -A monitoring interface shows the progress, logs and allow actions like pausing or stopping jobs. - -The Oozie Editor/Dashboard application allows you to define Oozie -workflow, coordinator, and bundle applications, run workflow, -coordinator, and bundle jobs, and view the status of jobs. For -information about Oozie, see [Oozie -Documentation](https://oozie.apache.org/docs/). - -A workflow application is a collection of actions arranged in a directed -acyclic graph (DAG). It includes two types of nodes: - -- Control flow - start, end, fork, join, decision, and kill -- Action - Jobs - -A coordinator application allows you to define and execute recurrent and -interdependent workflow jobs. The coordinator application defines the -conditions under which the execution of workflows can occur. - -A bundle application allows you to batch a set of coordinator -applications. - - -## Workflows - -In the Workflow Editor you can easily perform operations on Oozie action -and control nodes. - -### Action Nodes - -The Workflow Editor supports dragging and dropping action nodes. As you -move the action over other actions and forks, highlights indicate active -areas. If there are actions in the workflow, the active areas are the -actions themselves and the areas above and below the actions. If you -drop an action on an existing action, a fork and join is added to the -workflow. - -- Add actions to the workflow by clicking an action - ![image](images/action.png) button and drop the action on the - workflow. The Edit Node screen displays. - 1. Set the action properties and click **Done**. Each action in a - workflow must have a unique name. - -- Copy an action by clicking the ![image](images/copy.png) **Copy** - button. - -1. The action is opened in the Edit Node screen. -2. Edit the action properties and click **Done**. The action is added - to the end of the workflow. - -- Delete an action by clicking the ![image](images/trash.png) button. -- Edit an action by clicking the ![image](images/edit.png) button. -- Change the position of an action by left-clicking and dragging an - action to a new location. - -### Control Nodes - -- Create a fork and join by dropping an action on top of another - action. -- Remove a fork and join by dragging a forked action and dropping it - above the fork. -- Convert a fork to a decision by clicking the - ![image](images/convert.png) button. -- To edit a decision: - 1. Click the ![image](images/edit.png) button. - 2. Fill in the predicates that determine which action to perform - and select the default action from the drop-down list. - 3. Click **Done**. - -![image](images/note.jpg) **Note**: worfklows.xml and their job.properties -cab also directly be selected and executed via the File Browser. - -## Schedules - -In Coordinator Manager you create Oozie coordinator applications and -submit them for execution. - -### Editing a Coordinator - -In the Coordinator Editor you specify coordinator properties and the -datasets on which the workflow scheduled by the coordinator will operate -by stepping through screens in a wizard. You can also advance to -particular steps and revisit steps by clicking the Step "tabs" above the -screens. The following instructions walk you through the wizard. - -1. Type a name, select the workflow, check the **Is shared checkbox** - to share the job, and click **Next**. If the Coordinator Editor was - opened after scheduling a workflow, the workflow will be set. -2. Select how many times the coordinator will run for each specified - unit, the start and end times of the coordinator, the timezone of - the start and end times, and click **Next**. Times must be expressed - as UTC times. For example, to run at 10 pm PST, specify a start time - of 6 am UTC of the following day (+8 hours) and set the Timezone - field to America/Los\_Angeles. -3. Click **Add** to select an input dataset and click **Next**. If no - datasets exist, follow the procedure in [Creating a - Dataset](#creatingDataset). -4. Click **Add** to select an output dataset. Click **Save - coordinator** or click **Next** to specify advanced settings. -5. To share the coordinator with all users, check the **Is****shared** - checkbox. -6. Fill in parameters to pass to Oozie, properties that determine how - long a coordinator will wait before timing out, how many - coordinators can run and wait concurrently, and the coordinator - execution policy. -7. Click **Save coordinator**. - -### Creating a Dataset - -1. In the Coordinator Editor, do one of the following: - - Click **here** in the Inputs or Outputs pane at the top of the - editor. - - In the pane at the left, click the **Create new** link. Proceed - with [Editing a Dataset](#editingDataset). - -### Editing a Dataset - -1. Type a name for the dataset. -2. In the Start and Frequency fields, specify when and how often the - dataset will be available. -3. In the URI field, specify a URI template for the location of the - dataset. To construct URIs and URI paths containing dates and - timestamps, you can specify the variables - ${YEAR},${MONTH},${DAY},${HOUR},${MINUTE}. For example: - hdfs://foo:9000/usr/app/stats/${YEAR}/${MONTH}/data. -4. In the Instance field, click a button to choose a default, single, - or range of data instances. For example, if frequency==DAY, a window - of the last rolling 5 days (not including today) would be expressed - as start: -5 and end: -1. Check the advanced checkbox to display a - field where you can specify a coordinator [EL - function](http://archive.cloudera.com/cdh4/cdh/4/oozie/CoordinatorFunctionalSpec.html). -5. Specify the timezone of the start date. -6. In the Done flag field, specify the flag that identifies when input - datasets are no longer ready. - - -## Bundles - -A bundle consists in a collection of schedules. - -### Creating a Bundle - -1. Click the **Create** button at the top right. -2. In the Name field, type a name. -3. In the Kick off time field, choose a kick off time. -4. Check the Is shared checkbox to allow all users to access the - workflow. -5. Click **Save**. The Bundle Editor opens. Proceed with [Editing a - Bundle](#editingBundle). - -### Editing a Bundle - -In the Bundle Editor, you specify properties by stepping through screens -in a wizard. You can also advance to particular steps and revisit steps -by clicking the Step "tabs" above the screens. The following -instructions walk you through the wizard. - -1. Click **Add** to select a coordinator that the bundle will kick off. -2. Choose the kick off time. The time must be expressed as a UTC time. - For example, to run at 10 pm PST, specify a start time of 6 am UTC - of the following day (+8 hours). -3. To share the bundle with all users, check the **Is shared** - checkbox. -4. Click **Next** to specify advanced settings or click **Save - bundle**. -5. Fill in parameters to pass to Oozie. -6. Click **Save bundle**. - - -# Contrib -Those modules are not active enough to be officially maintained in the core Hue but those -are pretty functional and should still fit your needs. Any [contribution](https://github.com/cloudera/hue/wiki) is welcomed! - -## SDK -Check the [SDK guide](../sdk/sdk.html#new-application) or contact the community about how to build your own custom app. - -## HBase Browser - -We'll take a look at the new [HBase Browser App](http://gethue.com/the-web-ui-for-hbase-hbase-browser). - -Prerequisites before using the app: - -\1. Have HBase and Thrift Service 1 initiated (Thrift can be configured) - -\2. Configure your list of HBase Clusters in -[hue.ini](https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L467) -to point to your Thrift IP/Port - - -![image](images/note.jpg) **Note**: With just a few changes in the [Python API](https://github.com/cloudera/hue/blob/master/apps/hbase/src/hbase/api.py), -the HBase browser could be compatible with Apache Kudu. - - -### SmartView - -The smartview is the view that you land on when you first enter a table. -On the left hand side are the row keys and hovering over a row reveals a -list of controls on the right. Click a row to select it, and once -selected you can perform batch operations, sort columns, or do any -amount of standard database operations. To explore a row, simple scroll -to the right. By scrolling, the row should continue to lazily-load cells -until the end. - -### Adding Data - -To initially populate the table, you can insert a new row or bulk upload -CSV/TSV/etc. type data into your table. - - -On the right hand side of a row is a '+' sign that lets you insert -columns into your -row - -### Mutating Data - -To edit a cell, simply click to edit inline. - -If you need more control or data about your cell, click “Full Editor” to -edit. - -In the full editor, you can view cell history or upload binary data to -the cell. Binary data of certain MIME Types are detected, meaning you -can view and edit images, PDFs, JSON, XML, and other types directly in -your browser! - -Hovering over a cell also reveals some more controls (such as the delete -button or the timestamp). Click the title to select a few and do batch -operations: - -If you need some sample data to get started and explore, check out this -howto create [HBase table -tutorial](http://gethue.com/hadoop-tutorial-how-to-create-example-tables-in-hbase). - - -### Smart Searchbar - -The "Smart Searchbar" is a sophisticated tool that helps you zero-in on -your data. The smart search supports a number of operations. The most -basic ones include finding and scanning row keys. Here I am selecting -two row keys with: - - - domain.100, domain.200 - - -Submitting this query gives me the two rows I was looking for. If I want -to fetch rows after one of these, I have to do a scan. This is as easy -as writing a '+' followed by the number of rows you want to fetch. - - - domain.100, domain.200 +5 - - -Fetches domain.100 and domain.200 followed by the next 5 rows. If you're -ever confused about your results, you can look down below and the query -bar and also click in to edit your query. - -The Smart Search also supports column filtering. On any row, I can -specify the specific columns or families I want to retrieve. With: - - - domain.100[column_family:]    - - -I can select a bare family, or mix columns from different families like -so: - - - domain.100[family1:, family2:, family3:column_a] - - -Doing this will restrict my results from one row key to the columns I -specified. If you want to restrict column families only, the same effect -can be achieved with the filters on the right. Just click to toggle a -filter. - - -Finally, let's try some more complex column filters. I can query for -bare columns: - - - domain.100[column_a] - -This will multiply my query over all column families. I can also do -prefixes and scans: - - -    domain.100[family: prefix* +3] - - -This will fetch me all columns that start with prefix\* limited to 3 -results. Finally, I can filter on range: - - - domain.100[family: column1 to column100] - - -This will fetch me all columns in 'family:' that are lexicographically -\>= column1 but <= column100. The first column ('column1') must be a -valid column, but the second can just be any string for comparison. - -The Smart Search also supports prefix filtering on rows. To select a -prefixed row, simply type the row key followed by a star \*. The prefix -should be highlighted like any other searchbar keyword. A prefix scan is -performed exactly like a regular scan, but with a prefixed row. - - - domain.10* +10 - - -Finally, as a new feature, you can also take full advantage of the -[HBase filtering](denied:about:blank)language, by typing your filter -string between curly braces. HBase Browser autocompletes your filters -for you so you don't have to look them up every time. You can apply -filters to rows or scans. - - - domain.1000 {ColumnPrefixFilter('100-') AND ColumnCountGetFilter(3)} - - -This doc only covers a few basic features of the Smart Search. You can -take advantage of the full querying language by referring to the help -menu when using the app. These include column prefix, bare columns, -column range, etc. Remember that if you ever need help with the -searchbar, you can use the help menu that pops up while typing, which -will suggest next steps to complete your query. - -## Sqoop 1 Importer -Iport data from relational databases to HDFS file or Hive table using Apache Sqoop 1. It enables us to bring large amount of data into the cluster in just few clicks via interactive UI. This Sqoop connector was added to the existing import data wizard of Hue. - -In the past, importing data using Sqoop command line interface could be a cumbersome and inefficient process. The task expected users to have a good knowledge of Sqoop . For example they would need put together a series of required parameters with specific syntax that would result in errors easy to make. Often times getting those correctly can take a few hours of work. Now with Hue's new feature you can submityour Sqoop job in minutes. The imports run on YARN and are scheduled by Oozie. This tutorial offers a step by step guide on how to do it. - -Learn more about it on the [Importing data from traditional databases into HDFS/Hive in just a few clicks -](http://gethue.com/importing-data-from-traditional-databases-into-hdfshive-in-just-a-few-clicks/) post. - -## Sqoop 2 Editor - -The Sqoop UI enables transfering data from a relational database -to Hadoop and vice versa. The UI lives uses Apache Sqoop to do this. -See the [Sqoop Documentation](http://sqoop.apache.org/docs/1.99.2/index.html) for more details on Sqoop. - -### Creating a New Job - -1. Click the **New job** button at the top right. -2. In the Name field, enter a name. -3. Choose the type of job: import or export. - The proceeding form fields will change depending on which type is chosen. -4. Select a connection, or create one if it does not exist. -5. Fill in the rest of the fields for the job. - For importing, the "Table name", "Storage type", "Output format", and "Output directory" are necessary at a minimum. - For exporting, the "Table name" and "Input directory" are necessary at a minimum. -6. Click **save** to finish. - - -### Running a Job - -There's a status on each of the items in the job list indicating -the last time a job was ran. The progress of the job should dynamically -update. There's a progress bar at the bottom of each item on the job list -as well. - -1. In the list of jobs, click on the name of the job. -2. On the left hand side of the job editor, there should be a panel containing actions. - Click **Run**. - -### Creating a New Connection - -1. Click the **New job** button at the top right. -2. At the connection field, click the link titled **Add a new connection**. -3. Fill in the displayed fields. -4. Click **save** to finish. - -### Editing a Connection - -1. Click the **New job** button at the top right. -2. At the connection field, select the connection by name that should be edited. -3. Click **Edit**. -4. Edit the any of the fields. -5. Click **save** to finish. - -### Removing a Connection - -1. Click the **New job** button at the top right. -2. At the connection field, select the connection by name that should be deleted. -3. Click **Delete**. - -NOTE: If this does not work, it's like because a job is using that connection. - Make sure not jobs are using the connection that will be deleted. - -### Filtering Sqoop Jobs - -The text field in the top, left corner of the Sqoop Jobs page enables fast filtering -of sqoop jobs by name. - - -## ZooKeeper Browser - - -The main two features are: - -- Listing of the ZooKeeper cluster stats and clients -- Browsing and edition of the ZNode hierarchy - - -ZooKeeper Browser requires the [ZooKeeper -REST](https://github.com/apache/zookeeper/tree/trunk/src/contrib/rest) -service to be running. Here is how to setup this one: - -First get and build ZooKeeper: - -
-git clone https://github.com/apache/zookeeper
-cd zookeeper
-ant
-Buildfile: /home/hue/Development/zookeeper/build.xml
-
-init:
-       [mkdir] Created dir: /home/hue/Development/zookeeper/build/classes
-       [mkdir] Created dir: /home/hue/Development/zookeeper/build/lib
-       [mkdir] Created dir: /home/hue/Development/zookeeper/build/package/lib
-       [mkdir] Created dir: /home/hue/Development/zookeeper/build/test/lib
-
-   ...
-
- -And start the REST service: - -
-cd src/contrib/rest
-nohup ant run&
-
- -If ZooKeeper and the REST service are not on the same machine as Hue, go -update the [Hue -settings](https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L581) -and specify the correct hostnames and ports: - -
-    [zookeeper]
-
-      [[clusters]]
-
-        [[[default]]]
-          # Zookeeper ensemble. Comma separated list of Host/Port.
-          # e.g. localhost:2181,localhost:2182,localhost:2183
-          ## host_ports=localhost:2181
-
-          # The URL of the REST contrib service
-          ## rest_url=http://localhost:9998
-
- -## Git - -A basic read only version is done [HUE-951](https://issues.cloudera.org/browse/HUE-951). - -
- - - -
- Documentation improvements? Feel free to edit this page on GitHub and send a pull request! +
- - - - - - - - - diff --git a/tools/kubernetes/helm/hue/templates/deployment-hue-api.yaml b/tools/kubernetes/helm/hue/templates/deployment-hue-api.yaml deleted file mode 100644 index 38b48ab919c..00000000000 --- a/tools/kubernetes/helm/hue/templates/deployment-hue-api.yaml +++ /dev/null @@ -1,31 +0,0 @@ -{{- if .Values.api.enabled -}} -apiVersion: apps/v1 -kind: Deployment -metadata: - name: hue-api - labels: - deployment: hue -{{ if .Values.monitoring.enabled }} - annotations: - "sidecar.jaegertracing.io/inject": "true" -{{ end }} -spec: - selector: - matchLabels: - pod: hue-api - replicas: {{ .Values.hue.replicas }} - template: - metadata: - labels: - pod: hue-api - spec: - containers: - - name: hue - image: gethue/compose:latest - imagePullPolicy: Always - ports: - - containerPort: 8005 - resources: - requests: - cpu: "25m" -{{- end -}} diff --git a/tools/kubernetes/helm/hue/templates/ingress-https-v1.yaml b/tools/kubernetes/helm/hue/templates/ingress-https-v1.yaml index a6c2f21614a..242b40539b5 100644 --- a/tools/kubernetes/helm/hue/templates/ingress-https-v1.yaml +++ b/tools/kubernetes/helm/hue/templates/ingress-https-v1.yaml @@ -45,26 +45,11 @@ spec: pathType: ImplementationSpecific path: / {{- end }} - {{- if .Values.api.enabled }} - - host: {{ .Values.api.domain }} - http: - paths: - - backend: - service: - name: hue-api - port: - number: 8005 - pathType: ImplementationSpecific - path: /(.*) - {{- end }} tls: - hosts: - {{ .Values.ingress.domain }} {{- range .Values.ingress.extraHosts }} - {{ . | quote }} {{- end }} - {{- if .Values.api.enabled }} - - {{ .Values.api.domain }} - {{- end }} secretName: letsencrypt-hue-prod-tls {{- end -}} diff --git a/tools/kubernetes/helm/hue/templates/ingress-https-v1beta.yaml b/tools/kubernetes/helm/hue/templates/ingress-https-v1beta.yaml index 78d1d680cca..64183c56e84 100644 --- a/tools/kubernetes/helm/hue/templates/ingress-https-v1beta.yaml +++ b/tools/kubernetes/helm/hue/templates/ingress-https-v1beta.yaml @@ -39,23 +39,11 @@ spec: servicePort: 80 path: / {{- end }} - {{- if .Values.api.enabled }} - - host: {{ .Values.api.domain }} - http: - paths: - - backend: - serviceName: hue-api - servicePort: 8005 - path: /(.*) - {{- end }} tls: - hosts: - {{ .Values.ingress.domain }} {{- range .Values.ingress.extraHosts }} - {{ . | quote }} {{- end }} - {{- if .Values.api.enabled }} - - {{ .Values.api.domain }} - {{- end }} secretName: letsencrypt-hue-prod-tls {{- end -}} diff --git a/tools/kubernetes/helm/hue/templates/service-hue-api.yaml b/tools/kubernetes/helm/hue/templates/service-hue-api.yaml deleted file mode 100644 index 18f74128550..00000000000 --- a/tools/kubernetes/helm/hue/templates/service-hue-api.yaml +++ /dev/null @@ -1,15 +0,0 @@ -{{- if .Values.api.enabled -}} -apiVersion: v1 -kind: Service -metadata: - name: hue-api - labels: - pod: hue-api -spec: - selector: - pod: hue-api - ports: - - name: hue-api - port: 8005 - type: NodePort -{{- end -}} diff --git a/tools/kubernetes/helm/hue/values.yaml b/tools/kubernetes/helm/hue/values.yaml index 8b6fbc04c9c..c275f9bd988 100644 --- a/tools/kubernetes/helm/hue/values.yaml +++ b/tools/kubernetes/helm/hue/values.yaml @@ -35,9 +35,6 @@ hue: celery_result_backend=redis://redis:6379/0 balancer: enabled: true -api: - enabled: false - domain: api.gethue.com workers: enabled: false replicas: 1