Dev to Main — March 2026

[maxachis]

Summary

Merges dev into main, collecting 8 PRs worth of features, fixes, and infrastructure improvements.

PRs Included

• feat(db): improve GET /annotate/all performance #601 — feat(db): improve GET /annotate/all performance
• Add auth guards to sensitive mutating endpoints #596 — Add auth guards to sensitive mutating endpoints
• Add prek-based pre-commit lint hook #595 — Add prek-based pre-commit lint hook
• Ensure federal agencies are linked only to US national location #594 — Ensure federal agencies are linked only to US national location
• Remove ReviewDog and add lint directives to CLAUDE.md #593 — Remove ReviewDog and add lint directives to CLAUDE.md
• Remove agency_described_not_in_database from data source metadata #591 — Remove agency_described_not_in_database from data source metadata
• fix(ia-save): validate save job status before recording success #590 — fix(ia-save): validate save job status before recording success
• feat: Internet Archive Collector #589 — feat: Internet Archive Collector

Issues Resolved

• Optimize annotation load time #566 — Optimize annotation load time
• New Collector: Internet Archive Crawler #379 — New Collector: Internet Archive Crawler
• Add admin permissions to sensitive endpoints #574 — Add admin permissions to sensitive endpoints
• Add pre-commit hook for linting #587 — Add pre-commit hook for linting
• Ensure Federal agencies are only linked to the US National Location #535 — Ensure federal agencies are only linked to the US national location
• Remove ReviewDog, add directives to CLAUDE.md #592 — Remove ReviewDog, add directives to CLAUDE.md
• Remove url_optional_data_source_metadata.agency_described_not_in_database property from data sources #514 — Remove agency_described_not_in_database property from data sources
• Solve Internet Archives Save Task Not Working #387 — Solve Internet Archives Save Task not working

Overall Changes

Performance

• Materialized url_annotation_count_view and url_annotation_flags views with unique indexes, yielding up to 92% improvement on GET /annotate/all at scale
• Added pytest-benchmark infrastructure with 10k-URL scale seeder, pyinstrument profiling, and CI job summary reporting

New feature

• Internet Archive collector that crawls the Wayback Machine CDX API to discover archived police data URLs, with filtering, preprocessing, and full test coverage

Security & auth

• Auth guards added to sensitive mutating endpoints (delete/put operations)
• Internet Archive save task now authorizes status polling requests and validates async save status before marking success

Developer experience

• Replaced ReviewDog with a prek-based pre-commit lint hook
• Benchmark CI workflow triggers on PRs to main
• Removed hardcoded dev DB password from docs in favor of docker-compose reference

Data model cleanup

• Removed agency_described_not_in_database from data source metadata
• Federal agencies are now enforced to link only to the US national location

🤖 Generated with Claude Code

[gitguardian]

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
15086721	Triggered	Generic Password	baf7550	.github/workflows/benchmark.yml	View secret
15086721	Triggered	Generic Password	ee09ea1	.github/workflows/benchmark.yml	View secret
15086721	Triggered	Generic Password	ee09ea1	.github/workflows/benchmark.yml	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}