Deploy the Sample App to Development

Expected time: 10–20 minutes after CI starts.

Outcome: The image built from an identified commit is running at https://dev.marsadlab.com/ and the GitOps overlay records the deployed revision.

This runbook deploys only sample-app to the sample-app-dev namespace. It does not deploy the main Marsad Web App, Model Server, or documentation site.

The release path is:

push to dev → GitHub Actions tests and publishes → select successful commit SHA
→ update and commit the dev overlay → apply with Kustomize → verify the rollout

The current repository does not run a GitOps controller. Committing the overlay preserves the audit trail, but kubectl apply performs the deployment.

Before You Start

You need:

• permission to push or merge the application change into dev;
• access to the repository's GitHub Actions runs and artifacts;
• kubectl configured for the Marsad development cluster;
• permission to update Deployments and PVCs in sample-app-dev; and
• a ready gateway-system/public Gateway, registry-credentials Secret, and shared-data-sample-app-dev static PV.

Run all local commands from the repository root. Confirm the cluster before changing it:

Set-Location F:\Projects\ByName\Marsad\Marsad
kubectl config current-context
kubectl auth can-i patch deployment/sample-app -n sample-app-dev
kubectl auth can-i create persistentvolumeclaims -n sample-app-dev
kubectl auth can-i patch persistentvolumeclaims -n sample-app-dev
kubectl -n sample-app-dev get secret registry-credentials
kubectl get pv shared-data-sample-app-dev
kubectl -n sample-app-dev get pvc shared-data --ignore-not-found -o yaml

The PV must be Available when no claim exists or Bound to sample-app-dev/shared-data. Stop if the context is wrong, authorization returns no, or either Ansible-owned prerequisite is missing.

1. Push the Application Change

Review and push the approved sample-app change to dev according to the team's branch policy:

git status --short
git branch --show-current
git push origin dev

The Sample App CI workflow runs automatically for pushes to dev that change sample-app/** or .github/workflows/sample-app.yml. A push that changes only another path does not publish a sample-app image. An authorized operator can also start the workflow manually on the dev branch from GitHub Actions.

2. Confirm the Image Was Published

In GitHub, open Actions → Sample App CI and select the run for the commit you intend to deploy. Confirm both jobs succeeded:

1. Test
2. Build and publish Docker image

The publish job pushes two tags to Marsad's private registry. Deployment uses the immutable tag:

registry.marsadlab.com/rafiulbiswas/marsad:dev-sha-<full-commit-sha>

Download the run artifact named sample-app-release-dev-<full-commit-sha> and inspect release-metadata.json. Confirm that:

• environment is dev;
• commit is the commit shown by the successful Actions run; and
• immutable_image ends in dev-sha-<the-same-full-commit-sha>.

Use the artifact's commit value as the release SHA. Do not use a shortened SHA, the mutable dev tag, a SHA from a failed run, or simply whichever commit is newest.

3. Update and Validate the Development Overlay

Open gitops/apps/sample-app/overlays/dev/kustomization.yaml and replace only the newTag value:

images:
  - name: registry.marsadlab.com/rafiulbiswas/marsad
    newTag: dev-sha-<full-40-character-commit-sha>

Validate the SHA and render the resulting manifest before applying it:

$releaseSha = '<full-40-character-commit-sha>'
if ($releaseSha -notmatch '^[0-9a-f]{40}$') { throw 'Release SHA must be 40 lowercase hexadecimal characters.' }

Select-String -Path gitops/apps/sample-app/overlays/dev/kustomization.yaml -Pattern 'newTag:'
kubectl kustomize gitops/apps/sample-app/overlays/dev | Select-String -Pattern 'image:'
kubectl kustomize gitops/apps/sample-app/overlays/dev | Select-String -Pattern 'kind: PersistentVolumeClaim|volumeName: shared-data-sample-app-dev|mountPath: /shared-data'
kubectl apply --dry-run=client -k gitops/apps/sample-app/overlays/dev

The rendered image must exactly match the immutable image recorded in release-metadata.json, and the overlay must contain its PVC and mount. If a shared-data claim already exists from the former Ansible ownership model, compare its access mode, volume mode, storage class, volume name, and requested capacity with the rendered claim. Stop if any immutable field differs; never delete the claim to force adoption.

4. Commit the GitOps Revision

Review and commit only the intended overlay update:

git diff -- gitops/apps/sample-app/overlays/dev/kustomization.yaml
git add gitops/apps/sample-app/overlays/dev/kustomization.yaml
git commit -m "chore: deploy sample app to development"

Push the commit directly or merge it through a pull request, according to the team's branch policy. Do not deploy an overlay revision that exists only in an operator's working tree. An overlay-only commit does not trigger Sample App CI because the workflow excludes gitops/** from its path filters.

5. Deploy Development

Apply the committed overlay, wait for the Deployment, and request the public endpoint:

Set-Location F:\Projects\ByName\Marsad\Marsad
kubectl apply -k gitops/apps/sample-app/overlays/dev
kubectl -n sample-app-dev get pvc shared-data
kubectl -n sample-app-dev rollout status deployment/sample-app --timeout=5m
curl.exe --fail --show-error https://dev.marsadlab.com/

The claim must report Bound to shared-data-sample-app-dev. A successful response contains "environment":"dev"; the command exits non-zero if HTTPS fails.

6. Verify the Exact Revision

Confirm that the Deployment and its Pods use the expected immutable image:

kubectl -n sample-app-dev get pvc shared-data
kubectl -n sample-app-dev get deployment sample-app -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl -n sample-app-dev get pods -l app.kubernetes.io/name=sample-app -o wide
kubectl -n sample-app-dev get httproute sample-app

The PVC must remain Bound, and the image output must end with dev-sha-<release SHA>. All desired replicas must be available, Pods must be Running and ready, and the route must be accepted.

If the Deployment Fails

Inspect the workload before retrying or changing the image:

kubectl -n sample-app-dev get pvc shared-data
kubectl -n sample-app-dev describe deployment sample-app
kubectl -n sample-app-dev get pods -l app.kubernetes.io/name=sample-app
kubectl -n sample-app-dev describe httproute sample-app

• A Pending claim means the Ansible-owned static PV is absent or its immutable binding contract differs from the GitOps claim. Do not delete either resource.
• A volume mount failure usually points to NFS, CSI, export, or node access.
• ImagePullBackOff usually means the tag was copied incorrectly, the publish job did not succeed, or registry-credentials cannot read the private image.
• A rollout timeout with a running container usually points to startup or readiness probe failures. Describe the failing Pod and inspect only its relevant recent logs.
• A successful rollout with a failed public request usually points to the HTTPRoute, Gateway, certificate, or DNS rather than the image build.

Roll Back

Revert the committed overlay update, push or merge that revert, and apply the development overlay again:

git revert <gitops-image-update-commit>
kubectl apply -k gitops/apps/sample-app/overlays/dev
kubectl -n sample-app-dev rollout status deployment/sample-app --timeout=5m
curl.exe --fail --show-error https://dev.marsadlab.com/

Rollback is complete only when the Deployment reports the previous immutable image and the public endpoint succeeds.