Senior Data Engineer at M1 Technology

  We are seeking a Senior Data Engineer to lead the development and optimization of data pipelines, data movement tools, and data preparation services to support advanced analytics and enterprise integration. This role requires deep expertise in handling both structured and unstructured data, designing scalable data architectures, and developing reliable ETL solutions. The ideal candidate has significant experience with both cloud-native and on-premise technologies, and thrives in collaborative, mission-driven environments. Clearance Requirement: TS/SCI with PolygraphKey Responsibilities:Design, implement, and maintain scalable data pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR, NiFi, and KubernetesPerform data ingestion, extraction, formatting, and transformation for a wide range of data types and formatsClean, preprocess, and format data for exploration, analysis, and operational useDevelop ETL tools and scripts and enhance existing pipelines using coding best practicesWork with Data Scientists, analysts, and mission partners to align data pipelines with mission goalsCollaborate with software engineers to configure and maintain data services and back-end componentsEnsure data quality, standardization, and provenance through collaboration with testing and data quality teamsProduce comprehensive documentation including ETL mappings, data lineage, code usage guides, and access instructionsParticipate in enterprise working groups to define and promote data standards across systemsSupport both one-time and ongoing data extraction and transformation efforts across various repositoriesRequired Qualifications:10+ years of experience in data lifecycle engineeringDeep expertise in ETL development, pipeline design, and data architectureStrong coding skills in Python, SQL, Spark, and other data engineering toolsExperience with COTS and open-source tools including ElasticSearch and Apache NiFiStrong understanding of cloud and on-premise data storage and processing solutionsExperience working in Agile development environmentsProven track record of integrating diverse data sources into enterprise systemsPreferred Qualifications:Experience deploying solutions on cloud platforms (e.g., AWS EMR, S3, RDS, EKS)Familiarity with data governance, data lineage tracking, and security requirementsHands-on experience with container orchestration using KubernetesKnowledge of data lake architectures and streaming data processingExcellent communication skills and experience collaborating with cross-functional teamsTech Stack & Tools:Languages & Frameworks: Python, SQL, SparkData Tools: Apache NiFi, Apache Iceberg, Trino, ElasticSearch, OpenSearchCloud & Infrastructure: AWS EMR, Kubernetes, Docker, On-Prem SystemsDevOps & CI/CD: Git, Agile methodologies, automated testing frameworksWhy Join M1?Tackle large-scale data engineering challenges with national impactWork with cutting-edge cloud and open-source data toolsCollaborate with a multidisciplinary team of engineers and analystsFlexible, mission-driven environment with opportunities for leadership  M1 Technology is an equal opportunity employer and values diversity. We do not discriminate in hiring on the basis of race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected by federal, state, or local law.