Repository navigation
[core] Fix the open Dependabot alerts: Django 5.2, Mako, protobuf, npm and more - #4391
Merged
Merged
Conversation
Update the npm dependencies behind the open Dependabot alerts. Direct dependencies: - qs 6.16.0 -> 6.16.1. - sanitize-html 2.17.7 -> 2.18.1. Requires Node >= 22.12. - i18next-http-backend 1.4.4 -> 4.0.6. It no longer has runtime dependencies and uses fetch. Hue only sets loadPath, which is unchanged. - webpack 5.104.1 and @babel/core 7.29.6 (dev). - Remove diff and @types/diff, which nothing imports. - Remove the jsdom 16.6.0 devDependency. Jest uses the jsdom that jest-environment-jsdom brings, and the unused copy pulled in the vulnerable @tootallnate/once 1.1.2. Transitive dependencies: - Add an override for prismjs 1.30.0, which refractor pins to 1.27. - Patch the vendored cuix tarball: dompurify 3.4.17, moment-timezone 0.5.48 and js-cookie 3.0.8. js-cookie 3 removed getJSON(), so useCookiePreference now parses the cookie itself. - Refresh the lock file for fast-uri 3.1.9 and serialize-javascript 7.1.4. Node: - Require Node >= 22.13 in package.json engines (eslint 9 and sanitize-html need it; CI already uses Node 22). - Install Node 22 in the container compile and base images, which still used Node 14, and update the docs. Verified with the full jest suite, tsc, and the main, login and worker webpack builds.
Bump the packages whose upgrade needs no code change: - python-ldap 3.4.3 -> 3.4.5 - requests 2.32.3 -> 2.33.0 - tablib 0.13.0 -> 3.10.0. It no longer installs xlrd, which the indexer needs for pandas.read_excel(engine="xlrd"), so pin xlrd==2.0.2 explicitly. Hue only uses tablib for CSV output. - PyJWT 2.14.0 -> 2.15.1 - djangorestframework-simplejwt 5.3.1 -> 5.5.1 - pyarrow 17.0.0 -> 23.0.1 - pytest 8.1.1 -> 9.0.3 (tests only) base_requirements.txt only exists so Dependabot can scan the pins, and it had drifted from generate_requirements.py (missing boto3, certifi, cryptography and pyOpenSSL). Rewrite it from the generator and add a test that fails when the two differ. The user, auth, filebrowser, indexer and desktop lib tests fail on the same 24 tests before and after the upgrade.
protobuf 3.20.3 is affected by two denial of service advisories (a JSON recursion depth bypass and a parsing issue) fixed in 5.29.x and 6.x. The checked-in signer_protos_pb2.py was old style generated code that does not import on protobuf 4 or newer. Regenerate it with protoc 6.33.5. The serialized file descriptor is byte for byte identical to the old one, so the wire format is unchanged. The generated module checks the runtime version, so the requirement is pinned to the same release. Add signer_protos_test.py, which round trips a SignRequestProto so a runtime mismatch fails a test instead of the first RAZ request.
Mako 1.2.3 is affected by three TemplateLookup path traversal advisories (double slash URI prefix, backslash URI and drive letter URI). The last one is only fixed in 1.4.2. djangomako 1.3.2, the Django template backend Hue uses for its Mako templates, hard pins Mako==1.2.3, so pip refuses the upgrade. There is no newer djangomako release. Vendor it under ext-py3 like the other patched libraries, unchanged except for relaxing its Mako requirement to >= 1.2.3, and install it as a local requirement. desktop/tests.py renders the full pages through the backend and passes on Mako 1.4.2.
The optional websocket results feature (the [[websockets]] configuration, the notebook consumer, the daphne ASGI entry point and the Helm deployment) is disabled by default, only streamed ksqlDB results through it, and has no tests. It keeps three packages (channels, channels-redis, daphne) on the dependency list that have to be kept compatible with every Django upgrade. Remove it instead, as the downstream branches already did: - Delete desktop/asgi.py, desktop/routing.py, notebook/routing.py and notebook/consumer.py, the websockets config section and has_channels(), and the channels settings. - Drop the WebSocket client code and the WEB_SOCKETS_ENABLED and WS_CHANNEL globals from the editor page and the frontend. Query results were already fetched over HTTP, which stays the only path. - ksqlDB queries no longer take a channel name and always return their rows synchronously. - Remove the daphne deployment, service and ingress route from the Helm chart. - Drop channels, channels-redis and daphne from the requirements. A [[websockets]] section left in an existing hue.ini is reported as an unknown section by the configuration check and otherwise ignored.
Django 4.2 reached the end of its security support in April 2026, so the Django advisories published since (GeoDjango/GDAL, DomainNameValidator header injection, cache middleware, signed cookie salts and more) are only fixed in 5.2 and later. djangorestframework 3.18, which fixes the remaining DRF advisories, also requires Django 5.2. Packages: - Django 4.2.30 -> 5.2.18, djangorestframework 3.15.2 -> 3.18.3, drf-spectacular 0.27.2 -> 0.29.0. - django-axes: replace the vendored 5.13.0 fork, which does not support Django 5, with 8.3.1 from PyPI. Remove the vendored fork and the unused django-babel fork. - django-webpack-loader 1.0.0 -> 3.2.3, django-auth-ldap 4.3.0 -> 5.3.0, djangosaml2 1.9.3 -> 1.12.0, django-celery-beat 2.6.0 -> 2.8.1, django-celery-results 2.5.1 -> 2.6.0, django-cors-headers 4.4.0 -> 4.9.0, django-extensions 3.2.3 -> 4.1, django-ipware 3.0.2 -> 7.0.1, django_prometheus 2.3.1 -> 2.4.1, django-redis 5.4.0 -> 6.0.0, lxml 4.9.1 -> 6.1.0, pytest-django 4.8.0 -> 4.12.0. Code changes: - settings.py: drop USE_L10N and the django_babel middleware and app. STATICFILES_STORAGE is ignored since Django 5.1, so configure the manifest static files storage through STORAGES to keep the hashed static file names. - django-axes 8: express the lockout options as AXES_LOCKOUT_PARAMETERS instead of the deprecated AXES_USE_USER_AGENT and AXES_LOCK_OUT_BY_COMBINATION_USER_AND_IP. AXES_BEHIND_REVERSE_PROXY and AXES_REVERSE_PROXY_HEADER no longer exist, which would have made every client behind the load balancer share one lockout. Map them to AXES_IPWARE_META_PRECEDENCE_ORDER. Remove a stale commented out middleware. - dt_logout: Django's LogoutView only accepts POST since Django 5.0, but Hue logs out through a link. Log the user out and redirect instead. - notebook/tasks.py: get_storage_class was removed, use import_string. - Pass the request context to render_bundle in the Mako templates, as django-webpack-loader 3 requires. - Run the database migration tests with the django_db_blocker fixture, as pytest-django 4.12 no longer allows them to open extra databases under the django_db mark. Add axes_lockout_test.py, which locks out a user after failed logins and checks that the lockout follows the forwarded client IP behind a proxy. The full test suite has the same failures before and after the upgrade (environment dependent oozie, jobbrowser, hive and RAZ tests).
|
✅ Test files were modified. Ensure that the tests cover all relevant changes. ✅ |
UI Code Coverage Report
|
CI runs `hue runruff check` on every file a PR changes, so the files the upgrade commits touched have to be lint clean as a whole. - Exclude generated *_pb2.py files from ruff. signer_protos_pb2.py is produced by protoc and has to be regenerated instead of edited. - auth/views.py: sort the imports, drop the unused OIDCBackend and Group imports and an unused exception binding. - notebook/tasks.py: sort the imports, drop the unused transaction and FileResponse imports and three unused local variables. - desktop/tests.py: sort the imports, drop the unused connection import and remove test_forbidden_libs, a Python 2 only test that always skipped. test_check_config_ajax now asserts on the decoded content it already computed, and test_cx_Oracle used the Python 2 only ex.message and an assert on a tuple, which can never fail. It now uses str(ex) and pytest.fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the open Dependabot alerts on
master(Python and npm) in six commits that can be reviewed one by one.Commits
diffandjsdom. The vendoredcuixtarball is patched (dompurify 3.4.17, moment-timezone 0.5.48, js-cookie 3.0.8; js-cookie 3 removedgetJSON(), souseCookiePreferencenow parses the cookie itself).engines.nodeis now>=22.13.0(sanitize-html and eslint 9 need it), the container images install Node 22, and the docs are updated.xlrd==2.0.2, which tablib no longer installs and the indexer needs), PyJWT 2.15.1, simplejwt 5.5.1, pyarrow 23.0.1, pytest 9.0.3.base_requirements.txt(only read by Dependabot) had drifted fromgenerate_requirements.py; it is regenerated and a test now fails if they differ.signer_protos_pb2.pyregenerated with protoc 6.33.5 (the serialized descriptor is byte for byte identical to the old one) plus a round trip test.Mako==1.2.3and has no newer release, so it is vendored underext-py3with that requirement relaxed.[[websockets]]section inhue.iniis reported as an unknown section and otherwise ignored.STORAGESinstead of the ignoredSTATICFILES_STORAGE, axes lockout options asAXES_LOCKOUT_PARAMETERS,behind_reverse_proxymapped toAXES_IPWARE_META_PRECEDENCE_ORDER(the old axes settings no longer exist, which would have made every client behind a load balancer share one lockout), GET logout withoutLogoutView,get_storage_classreplacement,render_bundlecontext in the Mako templates, pytest-django 4.12 migration test fixture. Removes the vendored django-axes and unused django-babel forks.Alerts not fixed
Testing
X-Forwarded-For.axes_lockout_test.py,signer_protos_test.py, the base requirements sync test.