Building an architecture dedicated to the industrialisation of Machine Learning raises many questions. Even more so when different Azure resources seem to play similar roles! Should I choose between Azure Machine Learning and Databricks? Or can the two interact?
From Databricks, I can redirect my logs to an MLFlow Tracker on Azure ML. With the azureml SDK, I can run code remotely on a Databricks cluster. How to choose according to my context? What are the best practices?
To make the right decisions, we need to understand who plays the storage, compute or versioning role and find the right candidate for each role. Several demonstrations will illustrate the possible advantages... without spending too much money!
We will end with a proposal for a multi-environment architecture, organised around the concept of "centralised registers".