{"id":15829,"date":"2019-01-03T15:09:00","date_gmt":"2019-01-03T23:09:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/migrating-scaling-machine-learning-models-to-azure-databricks-for-cloud-powered-ai-2\/"},"modified":"2023-12-11T13:34:43","modified_gmt":"2023-12-11T21:34:43","slug":"migrating-scaling-machine-learning-models-to-azure-databricks-for-cloud-powered-ai","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/migrating-scaling-machine-learning-models-to-azure-databricks-for-cloud-powered-ai\/","title":{"rendered":"Migrating & Scaling Machine Learning Models to Azure Databricks for Cloud-Powered AI"},"content":{"rendered":"

Needing to scale up your predictive power and data processing capabilities, but a bit apprehensive about moving awesome machine learning models to a new platform? No need to worry! In today’s post, I’ll show you the easy way to migrate and scale machine learning and deep learning\u00a0models from Python over to Azure Databricks. Plus, I’ll also talk about why reworking your existing models using MLlib<\/span> in Spark might be a good idea.<\/p>\n

\"MLlib<\/p>\n

Data scientists spend a lot of time training models and tuning them to optimize their performance for whatever use case is at hand. Traditionally, this is done on local workstations using machine learning libraries such as scikit-learn<\/span><\/a>,\u00a0H2<\/sub>0<\/span><\/a>, or\u00a0XGBoost<\/span><\/a> in Python.<\/p>\n

Many AI teams are making the shift to begin developing on Spark and Databricks, which allows for embarrassingly parallel model training, tuning, and cross-validation on a cluster. However, this doesn’t necessarily mean that we have to throw away the Python models that we’ve already built and have put to use. Migrating them to Databricks is easy!<\/p>\n

<\/p>\n

Migrating and Scaling from Python (scikit-learn)<\/h2>\n

Anyone who has used Python for machine learning has heard of scikit-learn<\/span><\/a>. It’s one of the most popular libraries for machine learning, consisting of a plethora of clustering, classification, regression, and dimensionality reduction algorithms.<\/p>\n

\"scikit-learn\"<\/p>\n

In 2016, the team at Databricks saw the need for users to be able to migrate their scikit-learn<\/span> models to Spark. Thus, they released the spark-sklearn<\/a><\/span> package.<\/p>\n

This package allows users to:<\/p>\n

    \n
  1. Train and evaluate models in parallel.<\/li>\n
  2. Spread the work across multiple machines with no changes required in the code between the single-machine case and the cluster case.<\/li>\n
  3. Convert Spark DataFrames seamlessly into Numpy<\/span><\/a> ndarrays or sparse matrices.<\/li>\n
  4. Distribute SciPy<\/span>‘s<\/a> sparse matrices as a dataset of sparse vectors.<\/li>\n<\/ol>\n

    \"spark-sklearn_flow\"<\/p>\n

    These features allow for better scalable processing of machine learning models without making users leave their scikit-learn<\/span> comfort zone.<\/p>\n\n\n\n
    \n
    from<\/span> sklearn import<\/span> svm, datasets\r\nfrom<\/span> spark_sklearn import<\/span> GridSearchCV\r\niris =<\/span> datasets.load_iris()\r\nparameters =<\/span> {'<\/span>kernel'<\/span><\/span>:('<\/span>linear'<\/span><\/span>, '<\/span>rbf'<\/span><\/span>), '<\/span>C'<\/span><\/span>:[1<\/span>, 10<\/span>]}\r\nsvr =<\/span> svm.SVC(gamma <\/span>= <\/span>'<\/span>auto'<\/span><\/span>)\r\nclf =<\/span> GridSearchCV(sc, svr, parameters)\r\nclf.fit(iris.data, iris.target)\r\n<\/code><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    In the example code above, you can see the use of both the normal scikit-learn<\/span> package to bring in the desired algorithm, as well as the spark-sklearn<\/span> package to perform a grid search across parameter and CV folds.<\/p>\n
    Scikit-learn<\/span> isn’t the only library that can be used in Databricks. Other packages such as H2<\/sub>O<\/span> and XGBoost<\/span> have Spark counterparts as well.\u00a0To learn how to use other third-party libraries in Databricks, click\u00a0here<\/a>.<\/p>\n
    Importing Trained Neural Networks (ONNX)<\/h2>\n
    ONNX, or the Open Neural Network Exchange, is a format for representing deep learning models such as neural networks. This format allows for users to train models on popular frameworks such as Cognitive Toolkit<\/span>, TensorFlow<\/a><\/span>, PyTorch<\/a><\/span>, and MXNet<\/a><\/span>,\u00a0and save them for distribution and use in other places.<\/p>\n
    \"ONNX\"<\/p>\n
    For example, let’s say we’ve created an awesome deep learning model on our local GPU-based workstation using Cognitive Toolkit<\/span>. Saving the model in the ONNX format is easy.<\/p>\n\n\n\n
    \n
    import<\/span> cntk<\/span> as<\/span> C<\/span>\r\n\r\nx<\/span> =<\/span> C<\/span>.<\/span>input_variable<\/span>(<\/span><<\/span>input<\/span> shape<\/span>><\/span>)<\/span>\r\nz<\/span> =<\/span> create_model<\/span>(<\/span>x<\/span>)<\/span> #your create model function<\/span>\r\nz<\/span>.<\/span>save<\/span>(<\/span><<\/span>path<\/span> of<\/span> where<\/span> to<\/span> save<\/span> your<\/span> ONNX<\/span> model<\/span>><\/span>,<\/span> format<\/span>=<\/span>C<\/span>.<\/span>ModelFormat<\/span>.<\/span>ONNX<\/span>)<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Once we’ve saved our model in a location accessible by Databricks (Blob Storage or Data Lake Store), we can import the model just as easily.<\/p>\n\n\n\n
    \n
    import<\/span> cntk<\/span> as<\/span> C<\/span>\r\nz<\/span> =<\/span> C<\/span>.<\/span>Function<\/span>.<\/span>load<\/span>(<\/span><<\/span>path<\/span> of<\/span> your<\/span> ONNX<\/span> model<\/span>><\/span>,<\/span> format<\/span>=<\/span>C<\/span>.<\/span>ModelFormat<\/span>.<\/span>ONNX<\/span>)<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    ONNX exporting and importing not only works with Cognitive Toolkit<\/span>, but a variety of other frameworks as well. For a list of tutorials on how to get started, click <\/span>here<\/a>.<\/span><\/p>\n
    In addition to using ONNX, you can also import from MLeap<\/a><\/span>, which is a common serialization format for machine learning pipelines. To learn how, click here<\/a>.<\/p>\n
    Retraining using MLlib<\/h2>\n
    Don’t forget that Spark includes a really powerful set of algorithms in MLlib<\/span>, Apache Spark’s scalable machine learning library. Personally, I’ve used MLlib<\/span> for quite a few clients here at BlueGranite and am starting to love it.<\/p>\n
    MLlib<\/span> includes the following classes of algorithms and functions:<\/p>\n