Repository navigation
[security] Pin certifi, upgrade pip/virtualenv and drop bundled CA lists (eMudhra) - #4389
Merged
Merged
Conversation
A security incident with eMudhra intermediate certificates required removing the emSign root CA from CA bundles such as certifi. Clients that keep trusting it are open to man-in-the-middle attacks. Hue builds installed an unpinned certifi (whatever was newest at build time) and shipped pip 22.x/25.x, which keeps trusting the compromised CAs through its vendored certifi regardless of the system trust store. Python packages: - generate_requirements.py: pin certifi==2026.10.23. 2026.10.19 is the first release without the compromised CA and 2026.10.23 also drops the remaining emSign roots. Add a test that fails if the pin is lowered below 2026.10.19. Build tooling: - Add desktop/core/pip-bootstrap-requirements.txt with pip 26.2.2 (the minimum fixed release) pinned by hash, and use it for the build host and for the virtual env created by `make virtual-env`. Hashes make pip verify the download regardless of which CAs the installing pip trusts. - Upgrade pip in the new virtual env: virtualenv seeds an older embedded pip. - Use virtualenv 21.14.13 on the build host (minimum fixed: 21.14.12) and stop installing it into the shipped env. Only virtualenv-make-relocatable is needed there and it does not import virtualenv. This also removes virtualenv, distlib, filelock and platformdirs from the shipped env. - Skip --copies on macOS: virtualenv >= 20.36 rejects it for Homebrew Python. - Drop the duplicated PIP_VERSION block in Makefile.vars. Packaging and images: - Cloudera builds: Python 3.11.18 (first 3.11 release that rejects the compromised intermediate CAs), pip and virtualenv versions in sync with the Makefile, and bootstrap get-pip.py with the pinned pip. - Dockerfiles: require pip >= 26.2.2 (and virtualenv >= 21.14.12 in tools/docker). - tools/container/build.sh: build with --no-cache --pull by default so images pick up the latest system certificates. Set DOCKER_BUILD_OPTS="" to opt out. Docs: note the Python 3.11.18 recommendation and the CA handling.
The vendored boto 2.49.0 shipped boto/cacerts/cacerts.txt, an unmaintained snapshot of a CA list, inside Hue's Python environment. It is only read when boto validates server certificates explicitly (https_validate_certificates=True, or validate_certs=True); Hue's own S3 connections pass validate_certs=None and use the platform trust store. A static CA list cannot follow CA removals such as the eMudhra remediation and trips CA-store scans. - Remove the boto.cacerts package and its packaging metadata (setup.py, MANIFEST.in). - connection.py: DEFAULT_CA_CERTS_FILE is now certifi.where(). Fall back to the system CA file when certifi is not installed (certifi is a dependency of requests, so it is always present in Hue). - Update the boto config tutorial that referred to the removed file. - Add aws/ca_bundle_tests.py: boto defaults to certifi's bundle, does not ship its own bundle, and certifi is not older than 2026.10.19.
tools/virtual-bootstrap held the Python 2 era virtual-bootstrap.py and its virtualenv_support wheels (pip 9.0.1, setuptools 28.8.0, wheel 0.29.0, argparse 1.4.0). Nothing runs it: virtual envs are created with virtualenv in `make virtual-env`, and the call in tools/relocatable.sh is commented out. The bundled pip 9.0.1 wheel also carries a 2016 CA bundle (pip/_vendor/requests/cacert.pem), which is exactly the kind of stale CA store that has to be kept out of the shipped tarball after the eMudhra CA incident. - Remove tools/virtual-bootstrap and tools/relocatable.sh (a manual script that only checked for virtual-bootstrap.py; relocatable envs are produced by `make relocatable-env` with tools/relocatable.py). - Makefile.tarball: stop shipping tools/virtual-bootstrap. - Makefile.vars.priv: drop the unused VIRTUAL_BOOTSTRAP variable. - Makefile: fix the build flow diagram. - tools/container/build.sh: drop the step that deleted the bundled wheels from the image.
|
✅ Test files were modified. Ensure that the tests cover all relevant changes. ✅ |
UI Code Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the eMudhra CA incident. The compromised eMudhra (emSign) CA was removed from CA bundles such
as
certifi, and clients that keep trusting it are exposed to man-in-the-middle attacks. Hue buildsinstalled an unpinned
certifiand shipped apipthat trusts the compromised CA through its ownvendored
certifi, regardless of the system trust store. See the python.org advisory for the minimumfixed versions: certifi 2026.10.19, pip 26.2.2, virtualenv 21.14.12, Python 3.11.18.
Changes
1. Pin certifi, upgrade pip and virtualenv
desktop/core/generate_requirements.py: pincertifi==2026.10.23(2026.10.19 is the first releasewithout the compromised CA; 2026.10.23 also drops the remaining emSign roots). A test fails if the pin
is lowered below 2026.10.19.
desktop/core/pip-bootstrap-requirements.txt: pip 26.2.2 pinned by hash, used bymake virtual-envfor the build host and for the new virtual env. The hashes make pip verify thedownload regardless of which CAs the installing pip trusts. virtualenv seeds an older embedded pip, so
the new env is upgraded explicitly.
Makefile/Makefile.vars: virtualenv 21.14.13 on the build host only. It is no longer installedinto the shipped env (only
virtualenv-make-relocatableis needed there and it does not importvirtualenv), which also drops virtualenv, distlib, filelock and platformdirs from the shipped env.
--copiesis skipped on macOS (virtualenv >= 20.36 rejects it for Homebrew Python). The duplicatedPIP_VERSIONblock is gone.tools/cloudera): Python 3.11.18 (first 3.11 release that rejects the compromisedintermediate CAs), pip and virtualenv versions in sync with the Makefile, and
get-pip.pybootstrapsthe pinned pip.
tools/docker).tools/container/build.sh: images are built with--no-cache --pullby default so they pick up thelatest system certificates. Set
DOCKER_BUILD_OPTS=""to opt out.2. Remove boto's bundled CA list
boto/cacerts/cacerts.txt, an unmaintained CA snapshot inside theHue environment. It is only read when boto validates certificates explicitly; Hue's own S3 connections
pass
validate_certs=None.boto.cacertspackage and its packaging metadata.DEFAULT_CA_CERTS_FILEnow usescertifi.where(), falling back to the system CA file if certifi is missing.aws/ca_bundle_tests.py: boto defaults to certifi's bundle, ships no bundle of its own, andcertifi is not older than 2026.10.19.
3. Remove unused
tools/virtual-bootstrap(separate commit, can be dropped)setuptools 28.8.0, wheel 0.29.0, argparse 1.4.0. Nothing runs it: virtual envs are created with
virtualenv, and the call in
tools/relocatable.shis commented out.tools/virtual-bootstrapandtools/relocatable.sh, and cleaned upMakefile.tarball,Makefile.vars.priv, the Makefile build-flow comment andtools/container/build.sh.Behavior changes
make virtual-envupgrades pip with--require-hashesfromdesktop/core/pip-bootstrap-requirements.txt.Bumping pip later means updating that file (version and sha256 digests).
DOCKER_BUILD_OPTS=""is set./opt/python/3.11.18.Test plan
Done:
Clean
make virtual-envon macOS (Apple Silicon, Python 3.11) with the new Makefile succeeded. Theresulting env has pip 26.2.2, certifi 2026.10.23 (only
emSign Root CA - G1left in its bundle), novirtualenv/distlib/filelock/platformdirs, no
boto/cacerts, and boto defaults to certifi's bundle.desktop/libs/awsanddesktop/core/generate_requirements_test.py: 42 passed.ruff checkpasses onthe changed files.
bash -npasses on the changed shell scripts. Commit titles passtools/ci/check_for_commit_message.sh.CI Python:
setup-python: '3.11'resolves to the newest 3.11 patch on the runner, which may be below3.11.18.