Staff Embedded Validation & Tools Engineer - OTA

As an Staff Embedded Validation & Tools Engineer - OTA, you'll own validation, automation, and operational readiness for firmware and software update delivery across our fleet of embedded devices. You'll design and run validation test programs that ensure OTA updates are robust, secure, and reliable under real-world conditions, including intermittent networks, power faults, device restarts, and other edge cases that matter to customers. You'll work closely with Firmware, Release, QA, and Cloud teams to close the loop from release to fleet health.

What You'll Do

• Design, implement, and execute comprehensive OTA validation plans across device families and release types (full OS images, firmware, bootloader, A/B updates, delta/patch updates)
• Build, manage, and scale automated testbeds and harnesses (hardware-in-the-loop, virtualized devices, emulators) to run regression, integration, and stress tests for OTA flows
• Implement automation for end-to-end OTA pipelines, from build artifact signing and staging to rollout and post-update verification, using Python, shell, and CI tools
• Validate update mechanisms and components, including the bootloader, update agent, package formats, rollbacks, A/B partitions, and recovery modes
• Test update resilience under adverse conditions such as intermittent networks (cellular/Wi-Fi), high latency/packet loss, power interruptions, low-battery scenarios, storage constraints, and corrupted images
• Validate security properties, including signing/verification, secure transport (TLS), secure boot, key management, and the vulnerability surface introduced by update components
• Define metrics and KPIs (success rate, rollback rate, mean time to recover, time-to-update), and instrument devices and cloud backends to measure rollout health and produce release readiness reports
• Perform root cause analysis on failed updates, reproduce field issues, and drive remediation with firmware and backend engineering teams
• Maintain device lab hardware (JTAG, serial consoles, network emulators, power controllers, logic analyzers) and tooling stacks (Wireshark, serial monitors, provisioning fixtures)
• Contribute to OTA process improvements, including staging strategies, canary rollouts, rollback thresholds, risk assessment templates, and runbook documentation

What You'll Bring

• Bachelor's degree in Electrical Engineering, Computer Science, or related field (or equivalent experience)
• 5+ years of hands-on experience validating embedded devices and firmware updates (OTA or similar)
• Strong programming/scripting ability in Python and familiarity with C/C++ for embedded debugging
• Deep understanding of embedded platforms (embedded Linux and/or RTOS), bootloaders (e.g., U-Boot), partition schemes, and A/B update models
• Practical experience building automated test harnesses and integrating tests into CI pipelines (Jenkins, GitLab CI, CircleCI, etc.)
• Experience with hardware debugging tools and workflows, including serial/UART, JTAG/SWD, logic analyzers, oscilloscopes, and power cycling rigs
• Familiarity with networking protocols used for OTA (HTTP(S), MQTT, CoAP) and wireless technologies (Wi-Fi, Ethernet, cellular, BLE), and experience analyzing logs, packet captures, and device telemetry to diagnose update failures
• Strong debugging skills and methodical problem solving, able to reproduce complex failure modes and propose design fixes, with excellent communication skills to write clear test plans, runbooks, and postmortems
• Preferred: Experience with cloud OTA platforms (AWS IoT Device Management, Mender, Azure IoT, Google Cloud IoT, or custom fleet management systems), secure update practices (code signing, PKI, TPM/secure element integration, secure boot), or CAN/LIN and automotive-grade validation
• Preferred: Familiarity with containerization/virtualization for test environments (Docker, QEMU), large-scale rollout strategies (canary/capacity ramping, staged rollouts, feature flags), firmware build systems (Yocto/OpenEmbedded, Buildroot), or telemetry frameworks and analytics (Prometheus, Grafana, ELK)

The salary for this position ranges from $140,000 - $170,000 per year, depending on experience, qualifications, and location.