# Data Engineer (Bid Role starting in 10 weeks) - TS/SCI w/ Poly **Company:** [Leading Path Consulting](http://jobs.workable.com/companies/bi8iDAazKi2YnMWSbhbr1B.md) **Location:** McLean, United States **Workplace:** on site **Employment type:** Full-time **Department:** Government [Apply for this job](http://jobs.workable.com/view/68698388-c7fb-44ed-bb1c-42edb3a59b92) ## Description The Sponsor requires Data Engineering support to evaluate, optimize, and implement robust data infrastructure that enables reliable, accessible, and scalable data delivery across the organization. The Contractor will work collaboratively with data consumers, technical teams, leadership, and stakeholders to assess current data pipelines, identify gaps in data accessibility and reliability, and architect solutions that establish trusted data foundations. Work involves applying engineering best practices to implement proper data modeling and integration patterns, ensuring data quality and observability throughout pipelines, and creating maintainable infrastructure that supports analytics, reporting, and operational use cases. The Sponsor's data landscape includes enterprise operational systems such as ServiceNow, network management platforms (NetIM), and network modeling tools (Forward Networks). The Data Engineering support must be adept at extracting data from these systems via APIs (Application Programming Interface), exports, and vendor-specific interfaces, often with limited documentation or non-standard data structures, and transforming this operational data into accessible, integrated datasets. WORK REQUIREMENTS: • The Contractor shall conduct comprehensive assessments of existing data pipelines, infrastructure, and data flows including integrations with operational systems like ServiceNow, network management platforms, and business applications to identify technical debt, bottlenecks, and reliability issues. • The Contractor shall evaluate current data architecture against industry best practices and organizational needs; develop technical recommendations and roadmaps for data infrastructure improvements. • The Contractor shall design, build, and maintain production-grade data pipelines using orchestration tools such as Airflow or Prefect. • The Contractor shall develop robust ETL (Extract-Transform-Load) / ELT (Extract-Load-Transform) processes from diverse sources: SaaS platforms, network management systems, databases, APIs, files, and streams. • The Contractor shall build API integrations handling authentication (OAuth, API keys, and Single Sign-On (SSO)), rate limiting, pagination, retry logic, and error handling. • The Contractor shall extract data from systems not designed for export; reverse-engineer undocumented data structures and relationships. • The Contractor shall handle semi-structured data (JSON and XML); and transform into structured datasets with consistent schemas. • The Contractor shall design dimensional models, data warehouses, and data marts following industry methodologies. • The Contractor shall create conceptual, logical, and physical data models optimized for query performance and storage efficiency. • The Contractor shall implement slowly changing dimensions and other data warehousing patterns. • The Contractor shall establish naming conventions, data standards, and modeling best practices. • The Contractor shall implement comprehensive data quality checks, validation rules, and automated monitoring with alerting. • The Contractor shall build error handling, failure recovery, logging, and observability into all processes. • The Contractor shall optimize pipelines for performance, cost, and resource utilization. • The Contractor shall develop reusable components and frameworks; refactor legacy pipelines for reliability. • The Contractor shall build and maintain data infrastructure on cloud platforms (Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)) using infrastructure-as-code using Terraform and CloudFormation. • The Contractor shall implement CI/CD pipelines, version control (Git), and automated testing frameworks. • The Contractor shall manage database performance tuning, indexing, partitioning, and capacity planning. • The Contractor shall establish backup, recovery, security controls, access controls, and compliance measures. • The Contractor shall partner with analysts, software developers, and business stakeholders to translate requirements into technical solutions. • The Contractor shall create comprehensive documentation for systems, processes, and integrations. • The Contractor shall provide technical guidance on data availability and proper usage; enable self-service access. • The Contractor shall troubleshoot pipeline failures, performance issues, and data discrepancies; perform root cause analysis. TRAVEL: Travel is anticipated for this contract: YES Travel is anticipated for this contractor upon Sponsor approval for the following categories: Local travel/POV will be on an as needed basis, within the local place of performance. Required Skills: 1\. Demonstrated experience building and managing data pipelines. Demonstrated experience with python. 2. Demonstrated experience with cloud computing using AWS services. Demonstrated experience processing data using Apache Spark. 3. Demonstrated experience with an RDBMS (Relational Database Management System) (Postgres, Oracle, MySQL) and writing SQL queries. 4. Demonstrated experience with Linux and shell scripting. 5. Demonstrated experience analyzing data in different file formats like csv, xml, json, avro, parquet, etc. 6. Demonstrated experience writing and validating unit tests. Desired Skills: 1\. Demonstrated experience in NiFi, Apache AirFlow, or an equivalent solution or tool for orchestrating data pipelines. 2. Demonstrated experience with java or scala. 3. Demonstrated experience administering an EMR/Spark cluster. 4. Demonstrated experience conducting performance tuning of a spark job. 5. Demonstrated experience supporting Hive, Iceberg, or another technology providing SQL access to data. 6. Demonstrated experience developing cloud-based security solutions. 7. Demonstrated experience following a configuration management process to review and deploy code as part of releases. ## Requirements **Active TS/SCI w/ FS poly required prior to application** Required Skills: 1\. Demonstrated experience building and managing data pipelines. Demonstrated experience with python. 2. Demonstrated experience with cloud computing using AWS services. Demonstrated experience processing data using Apache Spark. 3. Demonstrated experience with an RDBMS (Relational Database Management System) (Postgres, Oracle, MySQL) and writing SQL queries. 4. Demonstrated experience with Linux and shell scripting. 5. Demonstrated experience analyzing data in different file formats like csv, xml, json, avro, parquet, etc. 6. Demonstrated experience writing and validating unit tests. Desired Skills: 1\. Demonstrated experience in NiFi, Apache AirFlow, or an equivalent solution or tool for orchestrating data pipelines. 2. Demonstrated experience with java or scala. 3. Demonstrated experience administering an EMR/Spark cluster. 4. Demonstrated experience conducting performance tuning of a spark job. 5. Demonstrated experience supporting Hive, Iceberg, or another technology providing SQL access to data. 6. Demonstrated experience developing cloud-based security solutions. 7. Demonstrated experience following a configuration management process to review and deploy code as part of releases. ## Benefits Leading Path is an award-winning Information Technology and Management Consulting firm focused on providing solutions in process, technology, and operations to our government and Fortune 500 clients. We offer a professional and family friendly work environment with a strong work-life balance. Leading Path provides a comprehensive and competitive benefits package including fully paid medical/dental/vision premiums, generous PTO, 11 Paid Holidays, 6% 401K contribution, annual training and tuition reimbursement, SPOT Award bonuses, regular team events, opportunities for professional growth and advancement and much more!