Decoupling the monolithic Terraform module (#273): current tree structure + future vision — feedback wanted

[bhumi46]

Update (Oct 2026): implemented. The as-built tree structure, what changed from this proposal and why, and the 6 PRs (#406–#411) are in this comment. The original proposal is kept below unchanged.

---

Context

Issue #273 is decoupling the monolithic aws-resource-creation Terraform module into independently-stateful layers. #274–#277 are done on the fork branch linked below — not yet merged to develop, no PR opened yet; #278–#281 (Layer 3) turned out to already be decoupled a different way — see the table below. #352 and #353, linked under #273's sub-task checklist, extend the same effort to a datacenter provisioning target and a GoDaddy-capable DNS abstraction, respectively.

This discussion is about the repository tree structure only — current (as-built) vs. proposed (future) — asking whether the proposed structure will actually hold up.

For reference: work-in-progress on fork bhumi46/infra, branch GH-issue-273 — not yet opened as a PR to develop.

Current vs. future: what's different

	Current (as-built — docs/TREE_STRUCTURE.md)	Proposed (vision — docs/TREE_STRUCTURE_FUTURE.md)
aws-resource-creation monolith (security groups, EC2, EBS, Route53, IAM)	Two parallel implementations exist today, same pattern as the Layer 3 row below: the legacy monolith (still the default path whenever PROVISIONING_COMPONENT is left none) and 5 independent Terraform roots, each its own state — security (#274), compute (#275, EC2), storage (#276, EBS), dns (#277, Route53), iam. azure/gcp have no split at all yet — still single monolithic modules.	aws-resource-creation retires once Layer 3 also has no legacy path left (see row below); azure/gcp gain the same 5-way split (no issue filed for this part yet)
Layer 3 (nginx / rke2 / rancher-import / nfs / postgresql / activemq)	Two parallel implementations exist today: the legacy embedded Terraform provisioners (still the default path) and a newer top-level ansible/<component>/ tree, wired up via a dedicated CI configure job	Legacy provisioners retired; ansible/<component>/ becomes the only implementation, reused unchanged across every compute provider — Ansible runs over SSH and doesn't care who provisioned the host
Domain / DNS	Cloud-native only (Route53 for AWS)	A dns-providers/ abstraction — Route53 / Azure DNS / Cloud DNS / GoDaddy — selectable independently of compute provider (#353)
Inventory generation (Terraform → Ansible handoff)	One path: CI reads Terraform outputs + .tfvars, renders inventory.yml	Two paths: the existing CI/Terraform-driven one (unchanged) plus a new standalone, env-var-driven script for cases with no Terraform involved at all (e.g. pre-racked data-center hosts)
New: data-center as a provisioning target	Doesn't exist. ansible/inventory.example.yml covers configuring already-existing on-prem hosts, but there's no Terraform-equivalent that provisions a data center	A new datacenter compute-provisioning target, mirroring the aws layered split (#352) — this row has no "current" side, it's a net-new addition

• Full current-state directory map: docs/TREE_STRUCTURE.md
• Full proposed-future directory map: docs/TREE_STRUCTURE_FUTURE.md

Ask

Does the proposed structure above hold up — anything that looks structurally wrong, missing, or unnecessary before this becomes an implementation plan? Comments welcome on any row of the table.

[bhumi46]

Update — as-built tree structure (Oct 2026)

The proposal above has been implemented on branch GH-issue-273-v2 and raised as 6 stacked draft PRs into mosip/infra:GH-issue-273-v2: #406 → #407 → #408 → #409 → #410 → #411 (merge in order). This comment is the final structure; where it differs from the proposal, the reason is in the table below.

Goal it delivers: MOSIP infra deploys on AWS (Terraform + Ansible, fully automated) and on any pre-created VMs — data centre, Azure, GCP — with the same Ansible and no Terraform. Ships as a new feature release.

Final tree

infra/
├── profiles/                                  # deployment shapes, independent of where they run
│   ├── mosip/                                 # full platform (3 cp / 3 etcd / 1 worker)
│   │   ├── profile.yml                        # Layer-3 components, subdomains (single source), Ansible defaults
│   │   └── aws/{common,security,iam,compute,storage,dns,vm}.tfvars
│   ├── esignet-standalone/                    # same layout (1 / 1 / 2, no postgres/activemq)
│   └── observ/                                # same layout (1 node; Rancher UI + Keycloak) — was observ-infra
│
├── terraform/                                 # Layers 1–2: AWS provisioning
│   ├── base-infra/{aws,azure,gcp}/            # one-time VPC + WireGuard (azure/gcp: placeholders)
│   ├── modules/aws/                           # each with tests/main.tftest.hcl
│   │   ├── security/  iam/  compute/  storage/  dns/
│   │   └── instance-group/                    # generic EC2 + SG + IAM per workload
│   └── implementations/aws/                   # roots — one state each
│       ├── base-infra/
│       ├── security/  iam/  compute/  storage/  dns/   # applied in this order
│       └── vm/                                # standalone EC2 + SG + IAM, on demand
│
├── ansible/                                   # Layer 3: identical on AWS and on any VMs
│   ├── site.yml                               # ordered entrypoint
│   ├── requirements.yml  ansible.cfg
│   ├── inventory/
│   │   ├── generate.py                        # --from-terraform (AWS) | --from-hosts (data centre)
│   │   └── hosts.example.yml
│   ├── playbooks/<component>.yml              # run a single step
│   └── roles/{preflight,tls,nginx,rke2,rancher_import,nfs,postgresql,activemq,rancher_keycloak}/
│
├── .github/workflows/
│   ├── terraform.yml                          # COMPONENT = all | security | iam | compute | storage | dns | configure | vm | base-infra
│   ├── terraform-destroy.yml                  # reverse order
│   ├── terraform-component.yml                # reusable: plan/apply/destroy one root
│   └── infra-checks.yml                       # PR checks (fmt, terraform test, validate, generator tests, syntax)
│
└── docs/{DEPLOYMENT_SEQUENCE,PROFILES,DATACENTRE_DEPLOYMENT,TREE_STRUCTURE}.md

Retired: aws-resource-creation monolith, terraform/infra + terraform/observ-infra wrappers, implementations/aws/{infra,observ-infra}, the embedded provisioner modules (nginx-setup, rke2-cluster, nfs-setup, postgresql-setup, activemq-setup, rancher-keycloak-setup), azure/gcp infra placeholders, generate-ansible-inventory.sh.

How the two paths meet

AWS:          security → iam → compute → storage → dns ──► generate.py --from-terraform ─┐
Data centre:  pre-created VMs + hosts.yml ────────────────► generate.py --from-hosts ─────┤
                                                                                          ▼
              ansible/site.yml: preflight → tls+nginx → rke2 → rancher import → nfs → postgresql → activemq
                                (+ rancher/keycloak for observ)  →  Helmsman (unchanged)

Order is the legacy monolith's depends_on chain (IAM at EC2 creation, DNS before nginx). Preflight blocks nginx if DNS isn't ready for http01.

Proposed vs. built

Row in the proposal	Built
aws-resource-creation 5-way split, legacy path retired	✅ one root per component under implementations/aws/; legacy removed (#406, #408)
azure/gcp get the same 5-way split	❌ not built — their VMs are covered by the data-centre path; provisioning is a follow-up
Layer 3: ansible/<component>/ the only implementation	✅ converted to real roles under ansible/roles/ (the old defaults/main.yml were never loaded) + ordered site.yml (#407)
dns-providers/ (Route53 / Azure DNS / Cloud DNS / GoDaddy)	🔀 kept at modules/aws/dns (Route53), now multi-zone + extra_records (#409). Provider-agnostic TLS via tls_mode byo/http01/dns01 with any certbot plugin incl. GoDaddy (#407). Terraform GoDaddy records: not built (#353 open)
Two inventory producers	🔀 one generator with two inputs, same output — avoids two scripts drifting (#407)
datacenter as a Terraform provisioning target	🔀 data centre = pre-created VMs, no Terraform; hosts.yml + site.yml; guide in docs/DATACENTRE_DEPLOYMENT.md (#407, #411). Hypervisor provisioning not built (#352 open)
— (not in proposal)	➕ observ-infra collapsed into a profile; profiles/ at repo root (read by Terraform and Ansible)
—	➕ IAM before compute (legacy parity), vm component for standalone EC2 + SG + IAM (#410, new issue #405)
—	➕ CI: COMPONENT=all chains components in order; reverse-order destroy (the decoupled roots had none)

PRs (all draft, base GH-issue-273-v2)

PR	Issues
#406 decoupled roots + parity tests	closes #274–#277
#407 agnostic Ansible layer	closes #278–#281, #301
#408 one root per component, profiles, legacy retired, new CI	closes #282
#409 multi-zone DNS, zone-scoped certbot IAM	part of #353
#410 IAM before compute, vm component	closes #405
#411 data-centre guide, docs + diagrams aligned	part of #352

Verified: 52 terraform test runs, 7 roots validate, 14 generator unit tests, site.yml syntax per profile, all PR checks green. Not yet: a real AWS apply and a real data-centre run.

Feedback welcome on any row — especially Azure/GCP provisioning priority and whether #352/#353 should be closed as "covered by the agnostic path" or kept open for Terraform providers.