epic: Testing infrastructure and strategy

[joshrotenberg]

Overview

Consolidated tracking issue for test infrastructure, strategy, and improvements. Replaces #462, #568, #569.

Current Testing Map

Tests That Run in CI

Test Type	Location	Count	Mocking	CI Job
Unit tests	src/**/*.rs #[cfg(test)]	~100+	None	test-unit (parallel)
MCP tool tests	redisctl-mcp/tests/	41	wiremock	test-integration
CLI basic tests	redisctl/tests/cli_*.rs	384+	None (no API calls)	test-integration
Cloud output tests	redisctl/tests/cloud_output_test.rs	~10	wiremock	test-integration

Tests That Require Docker (Manual/--ignored)

Test Type	Location	Count	Requirement
docker-wrapper tests	redisctl/tests/docker_wrapper_tests.rs	10	Docker + Redis
Enterprise Docker tests	redisctl/tests/enterprise_docker_*.rs	44	docker-compose + RE
Root integration tests	tests/integration/*.rs	~20	docker-compose + RE

Coverage Summary

Component	Coverage	Notes
redis-cloud library	~95%	Comprehensive wiremock tests
redis-enterprise library	~100%	Comprehensive wiremock tests
redisctl-config	~76%	Config, credentials, profiles
CLI command wrappers	~0-30%	Thin glue code, low ROI to unit test
Workflows	0%	subscription-setup, init-cluster
Async utils	0%	Polling, timeout, retry logic
Support package	2%	Binary tar.gz handling
Overall	~24%	Libraries high, CLI glue low

---

Test Infrastructure

Mocking Stack:
├── wiremock 0.6 - HTTP server mocking
├── redis-cloud/testing - Cloud API fixtures & MockCloudServer
└── redis-enterprise/testing - Enterprise API fixtures & MockEnterpriseServer

CLI Testing:
├── assert_cmd 2.0 - Command execution & assertions
├── predicates 3.0 - Output matching
└── tempfile 3.8 - Isolated config directories

Container Testing:
├── docker-wrapper - RAII container lifecycle (Redis)
└── docker-compose - Redis Enterprise cluster

---

Testing Strategy

Philosophy

1. Libraries are the source of truth - redis-cloud and redis-enterprise have 95-100% coverage with wiremock mocks
2. CLI is thin glue - Command handlers mostly parse args → call library → format output. Low ROI to unit test.
3. Integration tests validate UX - assert_cmd tests cover argument parsing, help text, error messages
4. Real API tests are burn-in - Nightly tests against real Cloud/Enterprise catch integration issues

Burn-In Policy

Features are considered "validated" when they've passed:

1. Unit tests (if applicable)
2. Mocked integration tests
3. At least one nightly cycle against real API

Until a feature has been through a real API test cycle, it should be flagged in release notes as "not yet validated against production APIs."

---

Future Infrastructure

Phase 1: Self-Hosted Runner for Enterprise Tests

Goal: Run Enterprise Docker tests in CI nightly

name: Nightly Enterprise Tests
on:
  schedule:
    - cron: '0 3 * * *'  # 3am UTC
  workflow_dispatch:

jobs:
  enterprise-tests:
    runs-on: self-hosted  # Runner with Docker access
    steps:
      - uses: actions/checkout@v4
      - name: Start Redis Enterprise
        run: docker compose up -d && sleep 60
      - name: Run Enterprise tests
        run: cargo test --test enterprise_docker_integration_tests -- --ignored
      - name: Run docker-wrapper tests  
        run: cargo test --test docker_wrapper_tests -- --ignored
      - name: Cleanup
        if: always()
        run: docker compose down -v

Requirements:

• Self-hosted runner with Docker
• Redis Enterprise license for testing (or use trial)
• Secure secrets management

Phase 2: Cloud Test Account

Goal: Run real Cloud API tests nightly

name: Nightly Cloud Tests
on:
  schedule:
    - cron: '0 4 * * *'  # 4am UTC
  workflow_dispatch:

jobs:
  cloud-tests:
    runs-on: ubuntu-latest
    env:
      REDIS_CLOUD_API_KEY: ${{ secrets.CLOUD_TEST_API_KEY }}
      REDIS_CLOUD_API_SECRET: ${{ secrets.CLOUD_TEST_API_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - name: Run read-only Cloud tests
        run: cargo test --test cloud_integration_tests
      - name: Run write tests with cleanup
        run: cargo test --test cloud_write_tests -- --ignored
      - name: Orphan cleanup
        if: always()
        run: |
          # Delete any ci-test-* resources older than 1 hour
          cargo run -- cloud subscription list -o json \
            -q "[?starts_with(name, 'ci-test-')]" | ...

Requirements:

• Dedicated Cloud test account (not production)
• Spending limits/alerts configured
• API key with appropriate permissions
• Orphan resource cleanup job

Phase 3: Nightly Test Report

Aggregate results from both nightly jobs:

• Enterprise test pass/fail
• Cloud test pass/fail
• Coverage delta
• Any orphaned resources

---

Action Items

Documentation

• Create TESTING.md with test execution guide
• Document burn-in policy in CONTRIBUTING.md
• Add test badges to README

Infrastructure

• Set up self-hosted runner for Enterprise tests
• Request dedicated Cloud test account
• Create nightly workflow for Enterprise tests
• Create nightly workflow for Cloud tests
• Add orphan resource detector/cleanup

Test Coverage (High-Value Areas)

• Async utils (polling, timeout, retry) - high complexity
• Workflow orchestration - multi-step operations
• Support package system - binary handling
• Output formatting edge cases

Test Organization

• Consolidate test patterns across crates
• Standardize fixture usage
• Add test categories/tags for selective runs

---

Running Tests Locally

# Everything that runs in CI
cargo test --workspace --all-features

# Just unit tests (fast)
cargo test --lib --all-features

# Just CLI tests
cargo test --test cli_basic_tests --test cli_profile_tests

# Docker tests (requires Docker)
cargo test --test docker_wrapper_tests -- --ignored --nocapture

# Enterprise Docker tests (requires docker-compose up)
docker compose up -d
# Wait for cluster ready...
cargo test --test enterprise_docker_integration_tests -- --ignored

# With container reuse (faster iteration)
REUSE_CONTAINERS=1 cargo test --test docker_wrapper_tests -- --ignored

---

Closes

• Closes Test Coverage: Focus on High-Value Areas for v1.0 #462 (Test Coverage: Focus on High-Value Areas)
• Closes Cloud integration tests with CI secrets #568 (Cloud integration tests with CI secrets)
• Closes Documentation: Test suite structure and execution guide #569 (Documentation: Test suite structure)