Jobid=625455557961647618 (0.0989)
Senior Software Engineer – Data Quality Framework (PySpark / Databricks)
Location: Heerlen or Amsterdam, Netherlands (Onsite: 2 days per week after onboarding)
Hours per week: 40
Contract Type: Freelance / Interim Assignment (Not suitable for ZZP freelancers)
Start Date: ASAP
Duration: Initial contract with extension option
Maximum Hourly Rate: €42
Education Level: Bachelor's degree in Computer Science, Software Engineering, Data Engineering, or related field
Experience Required: Minimum 5 years
Travel Requirement: 25% – 50%
Language: English (Dutch is a plus depending on team setup)
Role Overview
We are seeking a Senior Software Engineer – Data Quality Framework to design, build, and maintain a scalable data quality framework within Databricks using PySpark.
You will play a key role in developing reusable framework components that support data validation, rule execution, and contract-driven data quality checks across enterprise data environments. The role combines strong software engineering practices with data platform engineering in a modern cloud-based ecosystem (Azure Databricks).
You will work in a multidisciplinary Agile team responsible for delivering reliable and scalable data capabilities that support critical business and data products.
Key Responsibilities
Data Quality Framework Development
- Design, develop, and maintain a PySpark-based data quality framework in Databricks
- Build reusable components for rule execution, validation logic, and result handling
- Implement contract-driven data validation mechanisms across datasets
- Develop scalable and maintainable framework architecture
Engineering & Platform Ownership
- Ensure high engineering standards through:
- Code refactoring and modular design
- Unit and integration testing
- CI/CD pipeline implementation and maintenance
- Packaging and versioning of framework components
- Support multi-environment deployment strategies (dev/test/prod)
Databricks Platform Operations
- Develop and manage Databricks jobs, workflows, alerts, and notifications
- Support operational stability of data pipelines and framework execution
- Contribute to monitoring and reliability improvements
Stakeholder Collaboration
- Work closely with data engineers, analysts, and platform teams
- Translate data quality requirements into scalable technical solutions
- Contribute to documentation, standards, and best practices
- Support coaching and knowledge sharing within the team
Assignment Deliverables
- A scalable and reusable data quality framework in Databricks
- Robust PySpark-based validation and rule execution components
- Stable CI/CD and deployment processes for framework components
- Improved reliability and governance of data quality processes
- Documented standards, guidelines, and reusable engineering patterns
- Operational monitoring setup (alerts, triggers, notifications)
Required Qualifications
- Minimum 5 years of experience in software engineering, data engineering, or platform engineering
- Strong proficiency in Python and PySpark
- Experience working with Apache Spark in production environments
- Hands-on experience with Databricks (Azure preferred)
- Strong understanding of:
- Software design principles (OOP, modular design, maintainability)
- CI/CD pipelines and testing strategies
- Multi-environment cloud deployments
- Experience with version control, packaging, and release processes
- Strong communication skills in English
- Ability to work independently and take ownership of tasks
Preferred Experience
- Experience building or maintaining data quality frameworks or rule engines
- Knowledge of metadata-driven validation approaches
- Experience with data contracts and governance concepts
- Familiarity with Databricks alerts, workflows, and operational tooling
- Experience with Azure DevOps and Artifactory
- Experience developing reusable platform components or shared libraries
- Exposure to enterprise-scale cloud data platforms
Key Competencies
- Strong software engineering mindset
- High attention to code quality and maintainability
- Analytical and structured problem-solving ability
- Ownership and proactive delivery attitude
- Ability to work across technical and business stakeholders
- Strong collaboration in agile teams
- Platform thinking and scalability focus
Deel deze vacature: