[docs] Refresh the docs and fix broken commands for the 4.12 release

README.md

@@ -16,105 +16,99 @@
 
 </div>
 
-## 🗄️ Overview
+## Overview
 
-Hue is a mature SQL Assistant for querying Databases & Data Warehouses, trusted by:
+Hue is an open-source SQL Assistant for querying databases and data warehouses. Use it to explore your data, write SQL with autocomplete, and share the results.
 
-* **1000+ customers** worldwide
-* **Top Fortune 500** enterprises 
+### Key features
 
-Organizations use Hue to quickly answer questions via self-service querying, executing hundreds of thousands of queries daily.
+- **Interactive SQL editor** with syntax highlighting and autocomplete.
+- **File Browser** to navigate and operate on HDFS, S3, ABFS, Ozone, and Google Storage (GS) files.
+- **Job Browser** to monitor and manage Hive queries, Impala queries, YARN applications, and Livy Spark jobs.
+- **Table Browser** to explore and manage database tables, schemas, and metadata.
+- **Table Importer** to create Hive and Impala tables from CSV and Excel files, uploaded from your computer or imported from a remote file system (HDFS, S3, ABFS, Ozone, GS).
+- **Multiple database connectors**, including Hive, Impala, MySQL, and PostgreSQL.
 
-### Key Features
+## Useful links
 
-- ✏️ **Interactive SQL editing** with syntax highlighting and autocomplete
-- 📁 **File Browser** for navigating and operating on HDFS, S3, ABFS, Ozone, and Google Storage (GS) files
-- 📊 **Job Browser** for monitoring and managing Hive queries, Impala queries, YARN applications, and Livy Spark jobs
-- 🗃️ **Table Browser** for exploring and managing database tables, schemas, and metadata
-- 📤 **Table Importer** for creating Hive and Impala tables from CSV/Excel files, with support for uploading from local system or importing from remote filesystems (HDFS, S3, ABFS, Ozone, GS)
-- 🔗 **Multiple database connectors** including Hive, Impala, MySQL, PostgreSQL, and more
-
-## 🔗 Useful Links
-
-- 🏠 **Website**: [gethue.com](https://gethue.com)
-- 🔌 **Connect to a database**: [Configuration Guide](https://docs.gethue.com/administrator/configuration/connectors/)
-- 🛠️ **Build your own Editor**: [SQL Scratchpad](https://docs.gethue.com/developer/components/scratchpad/)
-- ⚙️ **Query Service**: [Kubernetes Setup](https://docs.gethue.com/administrator/installation/cloud/#kubernetes) and [API](https://docs.gethue.com/developer/api/)
+- **Website**: [gethue.com](https://gethue.com)
+- **Documentation**: [docs.gethue.com](https://docs.gethue.com)
+- **Connect to a database**: [Connectors](https://docs.gethue.com/administrator/configuration/connectors/)
+- **Build your own editor**: [SQL Scratchpad](https://docs.gethue.com/developer/components/scratchpad/)
+- **Query service**: [Kubernetes setup](https://docs.gethue.com/administrator/installation/cloud/#kubernetes) and [API](https://docs.gethue.com/developer/api/)
 
 ![Hue Editor](https://cdn.gethue.com/uploads/2021/02/hue-4.9.png)
 
-## 🚀 Getting Started
-
-### Try Hue Now
+## Get started
 
-- 🌐 **Live demo**: [demo.gethue.com](https://demo.gethue.com/)
-- 🧪 **Quick integrations**:
+- **Live demo**: [demo.gethue.com](https://demo.gethue.com/)
+- **Quick integrations**:
   - [Hive](https://docs.gethue.com/administrator/configuration/connectors/#apache-hive)
   - [Impala](https://gethue.com/blog/quickstart-sql-editor-for-apache-impala/)
-  - [Trino/Presto](https://docs.gethue.com/administrator/configuration/connectors/#trino-presto)
+  - [Trino/Presto](https://docs.gethue.com/administrator/configuration/connectors/#trino)
   - [Phoenix SQL / HBase](https://gethue.com/blog/querying-live-kafka-data-in-apache-hbase-with-phoenix/)
   - [Spark SQL](https://gethue.com/blog/querying-spark-sql-with-spark-thrift-server-and-hue-editor/)
 
-Choose one of these deployment options to start the server, then [configure the databases](https://docs.gethue.com/administrator/configuration/connectors/) you want to query:
+Choose a deployment option to start the server. Then [configure the databases](https://docs.gethue.com/administrator/configuration/connectors/) you want to query.
 
-### 🐳 Docker
+### Docker
 
-Start Hue instantly:
+Start Hue:
 
 ```bash
 docker run -it -p 8888:8888 gethue/hue:latest
 ```
 
-Hue will be available at [http://localhost:8888](http://localhost:8888)
+Open [http://localhost:8888](http://localhost:8888). The first user to log in becomes the superuser.
 
-📖 See the [Docker Guide](https://github.com/cloudera/hue/tree/master/tools/docker/hue) or watch the [Quick Start Video](http://gethue.com/getting-started-with-hue-in-2-minutes-with-docker/)
+> **Note:** The image uses an embedded SQLite database by default. Connect Hue to MySQL or PostgreSQL before real use. See the [Docker guide](tools/docker/hue/README.md).
 
-### ☸️ Kubernetes
+### Kubernetes
 
 ```bash
 helm repo add gethue https://helm.gethue.com
 helm repo update
 helm install hue gethue/hue
 ```
 
-📖 Read more about configurations in the [Kubernetes docs](tools/kubernetes/)
+See the [Kubernetes guide](tools/kubernetes/README.md) and the [Hue Helm chart](tools/kubernetes/helm/hue/README.md) for the configuration options.
 
-### 💻 Development Setup
+### Build from source
 
-#### Quick Start with Docker
+Hue requires Python 3.11 and Node.js 22.13 or later.
 
-Use the [Dev Environment Docker](https://docs.gethue.com/developer/development/#dev-docker) for the fastest setup.
+1. Install the [dependencies](https://docs.gethue.com/administrator/installation/dependencies/) for your operating system.
+2. Clone the repository and build Hue:
 
-#### Manual Setup
+   ```bash
+   git clone https://github.com/cloudera/hue.git
+   cd hue
+   export PYTHON_VER=python3.11
+   make apps
+   ```
 
-1. Install [dependencies](https://docs.gethue.com/administrator/installation/dependencies/)
-2. Clone and build:
-
-```bash
-git clone https://github.com/cloudera/hue.git
-cd hue
-make apps
-build/env/bin/hue runserver
-```
+3. Start the development server:
 
-Hue will be available at [http://localhost:8000](http://localhost:8000)
+   ```bash
+   build/env/bin/hue runserver
+   ```
 
-📖 Read more in the [development documentation](https://docs.gethue.com/developer/development/)
+Open [http://localhost:8000](http://localhost:8000).
 
-## 🧩 Components
+For a development environment in Docker, see [Dev Docker](https://docs.gethue.com/developer/development/#dev-docker). For more details, see the [development documentation](https://docs.gethue.com/developer/development/).
 
-Hue offers several powerful components:
+## Components
 
-- **SQL Editor** - Interactive query interface
-- **SQL Parsers** - Syntax handling for multiple dialects
-- **REST/Python/CLI APIs** - Programmatic access to all functionality
+- **SQL Editor**: interactive query interface.
+- **SQL Parsers**: syntax handling for multiple dialects.
+- **REST and Python APIs**: programmatic access to Hue.
 
-📖 Learn about [components](https://docs.gethue.com/developer/components/) and [APIs](https://docs.gethue.com/developer/api/)
+Learn more about the [components](https://docs.gethue.com/developer/components/) and the [APIs](https://docs.gethue.com/developer/api/).
 
-## 🤝 Contributing
+## Contributing
 
-We welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) guide to get started.
+Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) to get started.
 
-## 📜 License
+## License
 
 [Apache License, Version 2.0](LICENSE.txt)

apps/beeswax/src/beeswax/management/commands/beeswax_install_examples.py

@@ -15,13 +15,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import os
 import csv
-import pwd
 import json
 import logging
+import os
+import pwd
 
-from django.core.management.base import BaseCommand
+from django.core.management.base import BaseCommand, CommandError
 from django.utils.translation import gettext as _
 
 from beeswax.conf import LOCAL_EXAMPLES_DATA_DIR
@@ -31,8 +31,8 @@
 from desktop.lib.exceptions_renderable import PopupException
 from desktop.models import Document2, Document2Permission
 from hadoop import cluster
-from notebook.models import MockRequest, _get_example_directory, import_saved_beeswax_query, make_notebook
-from useradmin.models import User, get_default_user_group, install_sample_user
+from notebook.models import _get_example_directory, import_saved_beeswax_query, make_notebook, MockRequest
+from useradmin.models import get_default_user_group, install_sample_user, User
 
 LOG = logging.getLogger()
 
@@ -42,9 +42,13 @@ class InstallException(Exception):
 
 
 class Command(BaseCommand):
-  args = '<hive|impala> <db_name>'
   help = 'Install examples but do not overwrite them.'
 
+  def add_arguments(self, parser):
+    # From the command line: beeswax_install_examples <hive|impala> [db_name]. Callers in the code use handle() with options.
+    parser.add_argument('args', nargs='*', metavar='dialect [db_name]',
+                        help='The dialect (hive or impala) and the database (default is "default").')
+
   def handle(self, *args, **options):
     if args:
       dialect = args[0]
@@ -55,6 +59,8 @@ def handle(self, *args, **options):
       self.tables = None
       interpreter = None
     else:
+      if 'dialect' not in options:
+        raise CommandError('Expected arguments: <hive|impala> [db_name]')
       dialect = options['dialect']
       db_name = options.get('db_name', 'default')
       interpreter = options.get('interpreter')  # Only when connectors are enabled. Later will deprecate `dialect`.
@@ -288,7 +294,7 @@ def load_partition(self, django_user, partition_spec, filepath, columns):
       partition_dir = self._get_partition_dir(partition_spec)
       hdfs_root_destination = self._get_hdfs_root_destination(django_user, subdir=partition_dir)
       filename = filepath.split('/')[-1]
-      hdfs_file_destination = self._upload_to_hdfs(django_user, filepath, hdfs_root_destination, filename)
+      self._upload_to_hdfs(django_user, filepath, hdfs_root_destination, filename)
 
       hql = LOAD_PARTITION_HQL % {'tablename': self.name, 'partition_spec': partition_spec, 'filepath': hdfs_root_destination}
 

apps/beeswax/src/beeswax/management/commands/close_commands_tests.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python
+# Licensed to Cloudera, Inc. under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  Cloudera, Inc. licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from io import StringIO
+from unittest.mock import MagicMock, patch
+
+import pytest
+from django.core.management import call_command, CommandError
+
+from beeswax.management.commands import close_queries, close_sessions
+
+COMMANDS = [
+  ('close_queries', close_queries.Command),
+  ('close_sessions', close_sessions.Command),
+]
+
+
+@pytest.mark.parametrize('name, command_class', COMMANDS)
+class TestCloseCommandsArguments:
+  def parse(self, name, command_class, argv):
+    return command_class().create_parser('hue', name).parse_args(argv)
+
+  def test_defaults_to_7_days_and_no_scope(self, name, command_class):
+    options = self.parse(name, command_class, [])
+
+    assert options.age_in_days == 7
+    assert options.scope is None
+
+  def test_age_in_days(self, name, command_class):
+    options = self.parse(name, command_class, ['0'])
+
+    assert options.age_in_days == 0
+    assert options.scope is None
+
+  def test_age_in_days_and_all(self, name, command_class):
+    options = self.parse(name, command_class, ['3', 'all'])
+
+    assert options.age_in_days == 3
+    assert options.scope == 'all'
+
+  def test_rejects_an_unknown_scope(self, name, command_class):
+    with pytest.raises(CommandError):
+      self.parse(name, command_class, ['3', 'everything'])
+
+  def test_rejects_a_non_numeric_age(self, name, command_class):
+    with pytest.raises(CommandError):
+      self.parse(name, command_class, ['week'])
+
+
+class TestCloseCommandsExecution:
+  def run_command(self, name, module, model, *args):
+    model_objects = MagicMock()
+    model_objects.filter.return_value = model_objects
+    model_objects.all.return_value = model_objects
+    model_objects.__iter__.return_value = iter([])
+    model_objects.__len__.return_value = 0
+    out = StringIO()
+
+    with patch.object(module, model) as model_class, patch.dict('os.environ', {'HIVE_CONF_DIR': '/tmp'}):
+      model_class.objects = model_objects
+      model_class.STATE = MagicMock()
+      with patch('beeswax.hive_site.reset'), patch('beeswax.hive_site.get_conf'), patch('beeswax.conf.HIVE_CONF_DIR'):
+        call_command(name, *args, stdout=out)
+
+    return out.getvalue()
+
+  def test_close_queries_with_the_age_and_all(self):
+    output = self.run_command('close_queries', close_queries, 'QueryHistory', '0', 'all')
+
+    assert 'Closing (all=True) HiveServer2 queries older than 0 days' in output
+
+  def test_close_queries_without_arguments(self):
+    output = self.run_command('close_queries', close_queries, 'QueryHistory')
+
+    assert 'Closing (all=False) HiveServer2 queries older than 7 days' in output
+
+  def test_close_sessions_with_the_age_and_all(self):
+    output = self.run_command('close_sessions', close_sessions, 'Session', '2', 'all')
+
+    assert 'Closing (all=True) HiveServer2 sessions older than 2 days' in output
+
+  def test_close_sessions_without_arguments(self):
+    output = self.run_command('close_sessions', close_sessions, 'Session')
+
+    assert 'Closing (all=False) HiveServer2 sessions older than 7 days' in output

apps/beeswax/src/beeswax/management/commands/close_queries.py

@@ -15,16 +15,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from builtins import str
 import logging
-from datetime import datetime,  timedelta
+from builtins import str
+from datetime import datetime, timedelta
 
 from django.core.management.base import BaseCommand
 
 from beeswax.models import QueryHistory
 from beeswax.server import dbms
 
-
 LOG = logging.getLogger()
 
 
@@ -37,25 +36,34 @@ class Command(BaseCommand):
   Closing (all=True) queries older than 7 days...
   0 queries closed.
   """
-  args = '<age_in_days> (default is 7)'
   help = 'Close finished Hive queries older than 7 days.'
 
+  def add_arguments(self, parser):
+    parser.add_argument('age_in_days', nargs='?', type=int, default=7,
+                        help='Close the queries older than this number of days (default is 7).')
+    parser.add_argument('scope', nargs='?', choices=['all'],
+                        help="Pass 'all' to close the queries whatever their state, not only the finished ones.")
+
   def handle(self, *args, **options):
-    days = int(args[0]) if len(args) >= 1 else 7
-    close_all = args[1] == 'all' if len(args) >= 2 else False
+    days = options['age_in_days']
+    close_all = options['scope'] == 'all'
 
     self.stdout.write('Closing (all=%s) HiveServer2 queries older than %s days...\n' % (close_all, days))
 
-    queries = QueryHistory.objects.filter(last_state__in=[QueryHistory.STATE.expired.value, QueryHistory.STATE.failed.value, QueryHistory.STATE.available.value])
+    queries = QueryHistory.objects.filter(last_state__in=[
+      QueryHistory.STATE.expired.value,
+      QueryHistory.STATE.failed.value,
+      QueryHistory.STATE.available.value,
+    ])
 
     if close_all:
       queries = QueryHistory.objects.all()
 
     queries = queries.filter(submission_date__lte=datetime.today() - timedelta(days=days))
 
     import os
+
     import beeswax
-    from beeswax import conf
     from beeswax import hive_site
     try:
       beeswax.conf.HIVE_CONF_DIR.set_for_testing(os.environ['HIVE_CONF_DIR'])