About this role
At Vanta, our mission is to help businesses earn and prove trust. We believe that security should be monitored and verified continuously, and we empower companies to practice better security and prove it with ease. Vanta has a kind and talented team, and while some have prior security experience, many have been successful at Vanta without it.
As a Staff Software Engineer on the Automation - Foundations team, you will lead the re-platforming of the data layer underneath Vanta’s entire compliance product.
Foundations ingests tens of thousands of security and compliance resources per second from across our customer’s environments. We store the data, catalog it, make it queryable, and turn it into evidence that has to survive a real SOC 2 or FedRAMP audit.
We are in the middle of moving our platform from a monolithic Mongo-centric architecture to a Kafka data pipeline that decouples data fetching from processing and is backed by Postgres and S3. Our biggest challenges are around supporting this migration at scale in a domain that requires correctness and while supporting an evolving set of requirements for upcoming products.
Visit our Vanta Engineering Blog https://www.vanta.com/all-categories/engineering to learn more about what our team is working on.
What you'll do as a Staff Software Engineer at Vanta:
- Lead the migration of Vanta’s resource data model from a Mongo-centric solution to a schema-aware Postgres-backed solution and running both generations in parallel without breaking a customer integration.
- Drive solutions across teams that you do not own but are dependent on the platform built by your team.
- Design for correctness under eventual consistency with idempotent session handling, conditional writes that survive out-of-order delivery, reconciliation rather than strict cross-service validation, and explicitly backpressure instead of implicit database limits.
- Design and evolve Vanta's data ingestion and pipeline architecture, ensuring reliable, high-throughput processing of terabyte-scale data streams across distributed systems
- Take our Query API from internal use to a production platform with schema versioning, joins and exports, per-tenant isolation, and predictable latency under load.
- Build data infrastructure where correctness is externally auditable as the evidence Vanta produces has to hold up in front of an auditor.
- Diagnose the failures that only appear at production data shapes and can include hot partitions, unbounded fan-out, and online rewrites of continuously-written tables.
- Set architectural direction for streaming infrastructure (Kafka and event queuing), caching layers (Redis), and persistent storage (Postgres, MongoDB) across the Foundations stack
- Mentor senior engineers across the team, raising the technical bar through design reviews, architectural guidance, and hands-on contributions to the most complex systems challenges
- Champion engineering excellence in reliabilit