Databricks Architect (Cosmos / Vector DB Engineer (AI Serving Layer)) \\Job Title: Databricks Data Architect\\ \\Location:\\ New Jersey (Hybrid) \\Responsibilities:\\ - Build and maintain ingestion frameworks (ADF / Databricks / Spark). - Implement Bronze to Silver transformations aligned to architecture. - Architect data quality checks, schema validation, and contract rules. - Optimize pipelines for performance and cost. - Own CI/CD pipelines (Azure DevOps). - Automate onboarding (repos, pipelines, policies, templates). - Maintain Infrastructure as Code (IaC) and scripting (Terraform, PowerShell / Python). - Troubleshoot pipeline failures and permission issues. - Design and implement curated data models in the Gold layer aligned to business use cases. - Build dimensional models (star/snowflake schemas), fact tables, dimensions, surrogate keys, and handle Slowly Changing Dimensions (SCD). - Ensure consistency across domains (common dimensions, conformed models), reusability and scalability of models, and alignment with the enterprise semantic layer (Power BI / Fabric / downstream tools). - Implement governance patterns defined by architecture. - Enforce Unity Catalog standards, naming conventions, metadata policies, and access controls (RBAC/ABAC). - Support delivery teams with onboarding, debugging, and design reviews. - Provide hands-on guidance and create reusable assets to reduce repeat questions. \\ Skills:\\ - \\Core Platform:\\ - Azure Data Platform (ADF, ADLS, Azure SQL). - Databricks (DLT, Delta Lake, Spark, Workflows, Unity Catalog). - Microsoft Fabric (preferred). - \\Engineering:\\ - Advanced PySpark / Python. - Strong SQL. - Pipeline design (batch + streaming). - \\DevOps / Automation:\\ - Azure DevOps (pipelines, repos, policies). - PowerShell automation. - CI/CD and Infrastructure-as-Code concepts. - \\Governance / Data Quality:\\ - Schema enforcement, data contracts, and data quality frameworks. - Understanding of metadata-driven design.