Skip to content

fix(helm): add liveliness and readiness probes to helm chart#143

Open
alukach wants to merge 4 commits intomainfrom
fix/140/add-k8s-health-probes
Open

fix(helm): add liveliness and readiness probes to helm chart#143
alukach wants to merge 4 commits intomainfrom
fix/140/add-k8s-health-probes

Conversation

@alukach
Copy link
Member

@alukach alukach commented Mar 9, 2026

What I'm changing

This PR attempts to integrate our existing /healthz endpoint into K8s environments

How I did it

I admittedly know very little about Helm/K8s and reached for Claude Code (Opus 4.6) to complete this. Please review thoroughly

closes #140

@github-actions github-actions bot added the fix label Mar 9, 2026
@alukach alukach force-pushed the fix/140/add-k8s-health-probes branch from b88b60f to 3d046b5 Compare March 9, 2026 04:51
Copy link

@thenav56 thenav56 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alukach, few concerns from my side.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to enable the health endpoint?

Currently, when running this inside a stac-auth-proxy pod container, it returns a 404:

>>> import httpx
>>> httpx.get("http://localhost:8000/healthz")
<Response [404 Not Found]>

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The healthz path DOES include the root_path, are you using that? ie does http://localhost:8000/stac/healthz work?

@thenav56
Copy link

thenav56 commented Mar 9, 2026

Note

Findings from a live environment test (tested in a separate internal cluster):

Config used

    - repoURL: https://github.com/developmentseed/stac-auth-proxy.git
      targetRevision: v1.0.3-rc2
      path: helm/
      helm:
        valuesObject:
          env:
            UPSTREAM_URL: "http://ifrcgo-montandon-eoapi-{{ $i }}-stac:8080"
            OIDC_DISCOVERY_URL: "https://goadmin-stage.ifrc.org/o/.well-known/openid-configuration"
            OVERRIDE_HOST: "0"
            ROOT_PATH: "/stac"
            COLLECTIONS_FILTER_CLS: stac_auth_proxy.montandon_filters:CollectionsFilter
            ITEMS_FILTER_CLS: stac_auth_proxy.montandon_filters:ItemsFilter
          ingress:
            enabled: "true"
            host: "montandon-eoapi-{{ $i }}-auth-proxy.ifrc-go.dev.togglecorp.com"
            className: "nginx"
            tls:
              enabled: "false"
          replicaCount: 1
          extraVolumes:
            - name: filters
              configMap:
                name: stac-auth-proxy-filters
          extraVolumeMounts:
            - name: filters
              mountPath: /app/src/stac_auth_proxy/montandon_filters.py
              subPath: montandon_filters.py
              readOnly: true

When setting ROOT_PATH=""

  • When OIDC_DISCOVERY_URL is down
    • The pod does not exit or retry at the application level (same as before).
    • After some time, Kubernetes deletes the pod due to the health check.
    • Pod status is green after OIDC_DISCOVERY_URL is up
  • When OIDC_DISCOVERY_URL is up
    • Pod status is green after few seconds

When setting ROOT_PATH="/stac"

  • The pod gets stuck in an error state.
    • logs shows 404 for /healthz

How should the health check be handled when ROOT_PATH is defined?

@alukach
Copy link
Member Author

alukach commented Mar 9, 2026

The pod does not exit or retry at the application level (same as before).

@thenav56 Am I correct that even though you're using the newer helm chart, that helm chart is still pointing at the latest docker image which is still using v1.0.2 (link):
image

Perhaps we should hardcode this value to the helm chart version?

image:
repository: ghcr.io/developmentseed/stac-auth-proxy
pullPolicy: IfNotPresent
tag: "latest"

@thenav56
Copy link

Hey @alukach, I was also wondering the same thing for the image tag

After including this, everything is working as expected:

image

Would it be feasible to expose the health endpoint at /healthz instead of $ROOT_PATH/healthz? This would avoid requiring additional configuration changes on the consumer side.

It was also a bit confusing since all requests under /stac are redirected to the STAC API server, but the /stac/healthz endpoint is not.

If the chart consumer need to include that, then we should also update the documentation and note this as a breaking change in the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Helm: add Liveness/Readiness for stac-proxy deployment

3 participants