Why Open: Client is building out new team and initiative
Location: US- Remote, must work EST, east coast travel occasionally for sessions to client (paid for by softserve) maybe 1x per quarter, Louisville kentucky
MUST-HAVE
- 6+ years in data engineering roles + has been a tech lead before
- Very strong hands on experience with data bricks
- Expert with data pipeline development
- Strong Python (idiomatic, tested, production-grade)
- Hands-on Confluent Kafka — producers, consumers, Connect, Schema Registry
- Kubernetes experience (deployments, namespaces, resource management)
- Solid SQL and experience with columnar/analytical stores (BigQuery, Synapse)
- Comfort operating in multi-cloud environments (GCP + Azure)
- Understanding of data stream processing concepts (offsets, partitions, exactly-once semantics), constant movement of data
- Has done data integration work
- Strong with CDC- change data capture , concept, tools that support that (how you handle dataintegration into databricks)
- Someone who has an interest in being Ai driven, large amounts of data streaming between systems
- Very good communication + can speak with client
Industry:
- Insurance/healthcare insurance background
DAY TO DAY:
Our client, a large international consulting firm is seeking a Senior/Lead Data Engineer to join their team to support their health insurance client. The client has a new ai initiative, they are transforming their platform using ai, and the data team is looking at data sets and making sure the data is good quality, in the correct formats, centralized properly to feed into ai systems. We're looking for a Senior Data Engineer to design, build, and operate our real-time data infrastructure. You'll be at the center of our event-driven architecture — owning Confluent Kafka pipelines deployed on Kubernetes across GCP and Azure, and working closely with data scientists, ML engineers, and product teams to move data reliably at scale.
Design and support real-time data streaming solutions using Kafka and Confluent technologies, ensuring scalable and reliable data processing across enterprise platforms. Manage Kafka infrastructure on Kubernetes, including performance tuning, monitoring, capacity planning, and production support.
Develop Python-based data pipelines and streaming applications that enable efficient data ingestion, transformation, and delivery. Integrate data platforms across Google Cloud and Microsoft Azure environments to support analytics and business operations.
Maintain data quality, schema governance, and service-level objectives for both streaming and batch workloads. Collaborate with DevOps teams to automate deployments and infrastructure management through CI/CD pipelines and infrastructure-as-code practices.
Implement monitoring, alerting, and observability solutions to improve platform reliability and operational visibility. Partner with stakeholders to define data contracts, manage schema evolution, and ensure seamless integration with downstream applications and services.