Principal Data Platform Architect
We are seeking a Principal Data Platform Architect with proven experience leading enterprise data platform migrations from Palantir Foundry to Databricks.
This role will drive the strategic design, migration, and optimization of the enterprise data ecosystem, transitioning data pipelines, ontologies, and transformation workflows from Palantir Foundry to a scalable Databricks Lakehouse architecture.
The successful candidate will provide architectural leadership and technical direction, ensuring data integrity, governance, performance, and long-term platform sustainability. You will collaborate closely with data engineering, DevOps, and business stakeholders to modernize the data platform, establish best practices, and deliver a secure and high-performance migration.
Responsibilities
Define and execute the end-to-end migration roadmap from Palantir Foundry to Databricks, including target architecture, data models, security, and governance frameworks.
Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL.
Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process.
Optimize Spark workloads, cluster configurations, and storage strategies to ensure high performance and cost-efficient operations in Databricks.
Collaborate with data engineers, platform teams, and business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability.
Skills
Must have
Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning.
Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline conversion, data model redesign, and production cutover.
Expert-Level PySpark & Python Skills. Strong ability to design, optimize, and refactor distributed data processing workflows.
Advanced SQL & Data Modelling Expertise. Experience in dimensional modeling, lakehouse architecture patterns, and query optimization.
Cloud Platform Experience (Azure preferred). Hands-on experience deploying and managing data platforms in cloud environments, including storage, security, and networking considerations.
Nice to have
Hands-on Expertise in Palantir Foundry. Proven experience with Foundry pipelines, ontologies, data lineage, transformations, and platform governance.
Understanding of the Investment Data Domain.
Familiarity with Dynatrace or Datadog for system observability and monitoring.
Databricks certification, cloud certifications (Azure/AWS), or enterprise data architecture certifications.
Other
Languages
English: C1 Advanced
Seniority
Senior
Recommended Jobs
Strategy Director - Loyalty Consulting
Collinson is the global leader in travel experiences and customer engagement, a business that builds loyalty 500 million times a day, across 140 countries, for some of the world's most ambitious bran…
HR Administrator - Recruitment & Compliance - Ealing
HR Administrator – Support Staff Recruitment and Statutory Compliance – Ealing A well-organized academy in Ealing is seeking an experienced and meticulous HR Administrator to provide comprehen…
Science Technician - Grammar School, Redbridge
Science Technician – Grammar School, Redbridge Start Date: As soon as possible Contract: Full-time, Permanent Salary: Paid to scale School information: This is a highly regarded 11 …
Psychology Specialist | Islington Independent School
Tired of behavior management taking up your evenings? My client in Islington offers a scholarly environment where the focus is 100% on the science of the mind. This is a boutique Independent school l…
Account Manager - Agency
Company Overview Pulsar Group Plc is a tech innovator, delivering high quality SaaS products that address the fundamental business needs of clients in the marketing and communications industries. …
NHS HR Business Partner - Project Lead
Job Category : Interims Location : Town Hall, London Borough of Havering Hours Per Week : 36.00 Start Date : Immediate Start Start Time : 09:00 End Time : 17:30 Salary: £2…
Require occasional babysitting for two children after school
Hello! I’m looking for a reliable babysitter to help with after-school care a few days a week. My kids are 6 and 9, and they usually finish school at 3:15 pm. The role would involve collecting them, p…
Year 1 Teacher | Richmond
Are you an enthusiastic and nurturing Year 1 Teacher looking for a full-time role in a Richmond primary rated Outstanding? This Year 1 Teacher vacancy starts in January 2026 and offers the chance to …
Category Buyer - Indirect | Food Services Organisation
This is an exciting opportunity, with clearly defined career pathways and progression opportunities to support you in your long-term career aspirations. This role is pivotal in implementing successfu…