Role Description

We are seeking a Data Engineer with strong Databricks expertise to support the NGP (Next Generation Platform) data ecosystem. The role focuses on building scalable data pipelines, optimizing Databricks jobs, and enabling high-performance data processing frameworks within a cloud-based architecture.

The candidate will work closely with Snowflake, Azure, and distributed data platforms to deliver robust, production-grade data solutions supporting financial analytics and reference data systems.

Databricks & Data Engineering

• Design, develop, and maintain Databricks-based ETL pipelines using Apache Spark.
• Build scalable data processing frameworks for ingestion, transformation, and validation
• Optimize Databricks jobs, clusters, and workflows for performance, reliability, and cost efficiency.
• Develop and manage job pipelines, scheduling, and orchestration in Databricks environments.

• Troubleshoot Databricks job failures, performance bottlenecks, and pipeline issues.
• Improve execution efficiency through cluster tuning, job parallelization, and batching strategies.
• Support production stability and L4-level troubleshooting for critical pipelines.
• The role requires close collaboration with cross-functional teams in an Agile environment to deliver data-driven solutions. Exposure to AI tools such as GitHub Copilot and modern development practices will be considered an added advantage

Core/Must have skills

• Strong experience in:
• Databricks + Apache Spark
• SQL / PySpark / Scala
• Hands-on with:
• ETL pipeline development and optimization
• Distributed data processing frameworks
• Experience with:
• Snowflake and data warehousing concepts
• Azure cloud platform
• Strong analytical and debugging skills in complex data systems

Good to have skills

Experience in:

• Delta Lake / Unity Catalog
• CI/CD pipelines (Harness / Azure DevOps)
• Data quality frameworks / validation pipelines
• Exposure to:
• Financial domain / performance analytics / reference data platforms
• Knowledge of AI-assisted development tools (e.g., GitHub Copilot)

Work Schedule

Regular shift

Keywords (If any)

Why this role is important to us

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at StateStreet.com/careers

Read our CEO Statement