Operations Support Engineer - Lead

Overview

Black Canyon Consulting (BCC) is looking for a Lead Operations Support Engineer to join our DevOps and infrastructure team to support application developers and operations at the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM), an institute of the National Institutes of Health. This opportunity is full time at the NCBI in Bethesda, MD and/or remote.

NCBI is part of the National Library of Medicine (NLM) at the National Institutes of Health (NIH). NCBI advances science and public health by providing free access over the Internet to biomedical literature and genomic data, making it one of the 400 top most-visited sites in the world. NCBI has a diverse staff of smart, talented, and deeply technical people who collaborate to build critically valuable services for researchers, physicians, educators, students, and the general public.

For example, NCBI develops and delivers PubMed, an index of over 29 million biomedical research abstracts; and maintains ClinicalTrials.gov, a database of over 400,000 clinical studies around the world.

The Systems & DevOps team is responsible for the efficient operation of infrastructure to run NCBI's many applications. This includes providing convenient, scalable solutions for development, deployment, and operations across teams, languages, and cloud and on-prem environments.

This is a great opportunity to work on challenging problems in a technical, scientific, and goal-oriented environment. NCBI offers flexible working hours, remote options, on- and off-site training courses, and conference attendance and tuition reimbursement.

Duties & Responsibilities

• NCBI has built a modern DevOps platform based on GitLab and Kubernetes, and is looking to create a team of support engineers to assist internal developers with transitioning legacy software development and deployment to the new DevOps platform. You would be the leader of our platform operations team.
• Identify and resolve operational problems in a micro-service environment
• Work with developers to resolve deployment and runtime problems
• Perform analysis and debugging work across multiple technologies
• Prioritize issues to keep applications within error budgets and meeting their SLOs
• Provide technical solutions to a wide range of problems and user requests
• Document process, procedures and SOPs by soliciting feedback and suggestions from team members
• Compile postmortems and action items to minimize future outages
• Interview other people for team member roles, and decide which ones to recommend for hire.
• Train new team members, and assist them with issues.

Provide on-call support to NCBI's internal developers and other staff.

Requirements

• BS degree in STEM or equivalent experience
• Customer-focused, team-oriented disposition
• Good systems debugging skills
• Comfortable with the Linux environment or UNIX CLI
• Experience with some programming or scripting language
• Have experience creating processes, procedures and SOP documentation
• General understanding of TCP/IP, HTTP, and related protocols
• Initiative to take ownership of tasks and drive them to completion
• Comfortable dealing with users with varying levels of IT knowledge
• Eager to learn new technologies
• Strong communication and soft skills to interface with customers, peers and management
• Good judgement, sense of integrity, and responsibility

Preferred Experience / Skillsets

• Kubernetes, OpenShift, Cloud or Linux experience
• Experience with:
• Service Reliability Engineering in any capacity
• Linux systems administration
• Automated CI servers, especially TeamCity and/or GitLab
• Automation programming/scripting in any of: bash, Ruby, Python, Go, Java, Scala, Rust, C++, Perl
• Automated configuration management, such as Puppet, Ansible, Chef, bcfg2, cfengine, etc. Puppet is preferred.
• Version control systems, especially git
• Service Mesh technologies (e.g., linkerd, Istio)
• Configuring or using monitoring and alerting technologies (TIGK stack, Grafana, Prometheus, OpsGenie)
• Confluence, Jira, and Microsoft Office suite
• GitOps tools, especially ArgoCD
• Google Anthos
• Understanding of:
• Linux internals (system calls, file systems, processes, etc.)
• Linux network configuration
• Linux application containerization, especially Docker
• Attached network storage technologies
• Cloud computing environment such as AWS, GCP or Azure
• Automated CI/CD pipelines
• Distributed systems design principles