About Etched

Etched is building hardware for frontier intelligence. We co-design chips, racks, software, and manufacturing to deliver best-in-class throughput and latency across both prefill and decode workloads. Our first products are heavily focused on inference. Backed by hundreds of millions from top-tier investors and staffed by leading engineers, Etched is redefining the infrastructure layer for the fastest growing industry in history.

Job Summary

You'll be one of Etched's first Site Reliability Engineers, building the command-and-control layer for our fleet of frontier inference systems: the alerting, automated recovery and fleet data that keep clusters serving, and the routing that gets each problem to the right expert. Our Supercomputing and Infrastructure teams build the fleet management, rollout and telemetry backbone; but you'll decide which signals define a healthy system, set how we alert and respond, and build agentic hardware health workflows on top of it.

Your job is to design and run a scalable system. Fleet SREs own detection, first response and handoff; domain SREs embedded in each team own the deep product fixes. Over time, most problems should be caught by an alert that measures what users experience, which is then triaged by agents, and sent to an engineer as needed, with the logs, dashboards and recent changes already attached.

Key Responsibilities

Define what a healthy inference system looks like. In collaboration with software teams building out management and telemetry systems, identify the signals that most clearly show an unhealthy site, rack, node or chip, keep refining them as the fleet changes, and decide which actions (drain, cordon, restart, reroute, roll back) run on their own and which page a person.

Set a common alerting standard. Define SLOs with each team (latency such as time to first token, error rate, availability), alert on user-facing symptoms, and verify that the metrics behind each alert measure what the alert assumes.

Cut alert noise. Suppress expected alerts during deploys and maintenance automatically, correlate related alerts into one incident, and retire pages nobody acts on.

Automate recovery. Turn runbooks into agentic hardware health recovery: service restarts, node drain and traffic shift, spare swaps. Build it with guardrails (scoped permissions, rate limits, blast-radius checks, an audit trail)..

Route every event to the right engineer. Classify events by domain, page the right domain SRE with context attached, and dispatch on-site technicians for physical work such as tray swaps, reseats and leak response.

Shape the fleet data. Work with Supercomputing and Infrastructure, who build the telemetry pipeline, so it captures the key signals. Set the tracing standard (OpenTelemetry) so every service can be followed end to end, and report uptime and reliability trends that drive engineering and capacity decisions.

Run operationally excellent incident management. Own severity levels, incident channels, communications, postmortems and corrective-action tracking. Design on-call rotations, handoffs and onboarding so on-call is sustainable.

Raise the reliability bar for services. Review service designs with their owners for failure modes and graceful degradation before they ship, and partner with Supercomputing on canaries, rolling node-by-node deploys and graceful drain, so updates stop causing downtime.

Share on-call in a follow-the-sun rotation with SREs in other time zones.

Hands-on debug

Be the first line of defense. Acknowledge, size the user impact, open the incident, and decide fast between restart, drain, rollback and escalation.

Find where requests are stuck. Trace a latency or throughput regression through the serving stack (queueing, prefill, decode, handoffs, cache) to the stage that's backing up.

Triage hardware on live racks. Work through errors: chips dropping off PCIe, firmware panic resets, chip-to-chip link errors, HBM ECC errors and power faults; run recovery tools and decide when a node comes out.