[RFC] Proposal: Automated Container Optimization and Image Footprint Reduction for MOSIP 1.3.x+

[SankarGaneshb]

Summary
A spot check of the mosipid namespace on Docker Hub reveals significant image footprint growth in 1.3.x releases compared to 1.2.x predecessors. For instance, artifactory-server reaches ~1.78 GB and authentication-service (v1.3.0) reaches ~1.19 GB compressed—roughly double earlier releases.

This proposal outlines an automated, policy-driven pipeline to audit container images, profile runtime dependencies, eliminate redundant layer creation, and reduce unpacked image sizes by a targeted 60% to 70%.

Key Operational Targets

• Pod Startup & Autoscaling: Reduce heavy container startup delays during traffic spikes, targeting pod startup from 3–5 minutes down to under 60 seconds.
• Kubernetes Storage Efficiency: Lower local node-cache storage pressure across Dev, QA, Pre-Prod, and Production clusters where layer caching multiplies disk consumption.
• Vulnerability Surface Reduction: Shrink the CVE exposure surface by systematically pruning unused OS packages, build toolchains, and interactive shells.
• Patching Toil Reduction: Streamline ongoing CVE remediation and engineering triage hours through cleaner base image families and shared build workflows.

Proposed Architecture & Workflow
We propose integrating a 6-stage validation pipeline into MOSIP’s shared build workflows (mosip/kattu):

1. Discover: Automated registry scanning and size audits with exact digest and tag recording.
2. Trace: Profiling runtime system calls and classloader paths under load probes.
3. Slim: Stripping unneeded build tools, package caches, and redundant shells.
4. Scan: Comparing CVE surfaces before and after optimization using consistent scanners (e.g., Trivy, Grype).
5. Validate: Automated functional verification leveraging Spring Boot Actuator (/actuator/health) and MOSIP’s existing test suites (mosip/test-management, mosip/mosip-functional-tests).
6. Promote: Environment-gated deployments managed via Helm charts (mosip/mosip-helm) and install scripts (mosip/mosip-infra).

Proposed Pilot Scope & Initial Targets
To minimize risk, we propose a staged rollout starting with non-transactional and job-based components:

1. First Pilot (admin-ui):

• Problem: The image stands at ~1.2 GB unpacked because a single RUN layer installs npm, zip, unzip, and wget on top of Nginx without cleanup, whereas the served Angular site is only 8 MB.
• Solution: Retain only necessary runtime tools (wget, unzip), use --no-install-recommends, and purge apt lists. Projected reduction: ~85% cut (~175–185 MB) with zero JVM classloading risk.

2. Second Pilot (Spring Boot chown Layer Fix):

• Problem: A recursive chown -R command running immediately after a COPY instruction duplicates jar files as a new layer in 3 out of 9 audited images (e.g., resident-service adds an identical 377 MB layer; kernel-salt-generator and kernel-notification-service exhibit similar duplication).
• Solution: Replace with COPY --chown directives to prevent duplicate layer storage.

3. Core Services & Content Architecture:

• Address artifactory-server (~1.78 GB, driven largely by a single 1.9 GB content layer) and identity-critical engines (id-authentication, id-repository) only after successful Pre-Prod stress and scale testing.

Technical Risk Mitigations

• JVM Classloading: Mitigate static stripping risks for Spring Boot, Apache Camel, and Vert.x by executing runtime load-probe tracing and establishing strict reflection allow-lists.
• Compatibility Verification: Avoid unverified claims by enforcing automated health and API functional suites in Dev and QA prior to promotion.
• Traceability: Record original digests, trace reports, and slimmed digests for every build to ensure seamless auditability and rollback paths.

Request for Maintainer Feedback
We would love architectural feedback from the community and core maintainers on:

1. Whether maintainers prefer automated post-build minification or upstream lightweight base image adjustments (e.g., distroless or Alpine JRE).
2. Moving chart defaults from mutable tags (like 1.3.0) to immutable SHA-256 digests.
3. Endorsement to proceed with a non-intrusive profiling pilot on feature branches of mosip/kattu, mosip/mosip-helm, and mosip/mosip-infra. Looking forward to your thoughts and guidance!

MOSIP_Container_Optimization_Executive_Deck.pptx

MOSIP Container Optimization Technical Blueprint and Architecture Review.pdf