Custom models

[Jarathael]

Hello !
It's me again !
On the same project, I want to save my scikit-learn model along with the processing step + my optimized threshold while defining a predict function using all that.
I'm not sure I understand correctly how this works with mlflow and if we can define a custom mlflow model (different from the flavor if I understood correctly) for kedro-mlflow or if this is even possible.

Thank you in advance :)

[Galileo-Galilei]

Hi again! I have a fun fact: this use case is the very reason why I built kedro-mlflow at first, tracking parameters was originally just a cool addon feature.

I have no time right now but you can see he discussion in issue #16 and the documentation here.
There is also a tutorial to explain precisely how to do this: https://github.com/Galileo-Galilei/kedro-mlflow-tutorial.

If you are stuck or need further help, I'll be available tomorrow.

[Galileo-Galilei]

Yes, this is exactly what it does (actually it does more than that, but this is the core feature) :) I acknowledge that the API is complicated and should be simplified (it ranks high on my to do list). If it is not clear for you that it solves your problem, it likely means I should improve the documentation.

Validation of your workflow

From what you describe, I guess your workflow is the following (correct me if I'm wrong):

1. You have created a "training" pipeline. This train pipeline looks roughly like this :
   1. Several nodes to preprocess_data (clean NA, create new features, encode categorical data to numeric, encode labels to numeric...)
   2. Perform hyperparameter tuning and select the "best" model
   3. Calculate some metrics to evaluate the overall performance of all the models and especially the best one
   4. Calculate the optimal threshold for prediction
2. All the elements you need to reuse to predict on new data are persisted in the catalog.yml (your trained model, the fitted encoder, the optimal threshold).
3. You run this training pipeline with kedro run --pipeline=training, and now all the elements to be reused are saved on your disk.
4. Now you have created an "inference pipeline" which roughly looks like this:
   1. takes new data as input
   2. do the same preprocessing as during training (but do not fit the encoders again, use the existing ones)
   3. predict with the model to retrieve predictions
   4. post process these predictions: select the ones above optimal threshold calculated during training, eventually reencode them to something human readable (e.g. "spam", "not spam" instead of 0/1)
      Note: you say that you have only one node which does all these steps: it will obviously work in this simple case. However, it is likely a very bad practice (although I can't be sure without seeing your code) because I strongly suspect that it means you copy/paste some code (e.g. your preprocessing functions) between training and inference. There is an operational risk that one day you end up with a different preprocessing between the 2 pipelines, that is why the tutorial suggest a more robust method by sharing nodes between the 2 pipelines

At this stage, you can launch kedro run --pipeline=inference and get the predictions locally, but what you want now is something which is able to convert this inference pipeline to a mlflow model you can turn in an API, say something like:

# pseudo code, should be explicit
mlflow_model=convert_pipeline_to_model(your_inference_pipeline)
model_artifacts=extract_artifacts(your_inference_pipeline, your_catalog) # the artifacts is a dict containing all data need as input, like your model, your encoders, your optimal threshold...)

mlflow.pyfunc.log_model(mlflow_model, artifacts=model_artifacts)

And once your model is logged in mlflow, you can look at it in the UI, retrieve its run id, and serve it as an API with :

mlflow models serve -m "runs:/<your-model-run-id>/model"

When you send new data to this API, your entire pipeline is run (including preprocessing and postprocessing) and it returns your predictions.

Do you confirm that it is what you try to achieve?. If yes, the pipelines in the tutorial should be very close to what you have. I strongly suggest that you clone the repo and use kedro-viz to see how it is built.

Explanation of what kedro-mlflow does under the hood

Actually, kedro-mlflow does something very similar to what is described above. There is a KedroPipelineModel class which is a custom mlflow flavor (as you suggest) that can convert a Kedro pipeline to a "Mlflow model".
Then we can log it with mlflow.pyfunc.log_model(mlflow_model, artifacts=model_artifacts) and serve it as an API.

Actually, kedro-mlflow enforces an extra constraint: it does not let you save directly your inference pipeline to mlflow. Instead, it forces you to bind your 2 pipelines ("training" and "inference" above) into a single object with a function call "pipeline_ml_factory".

There are two main reasons why we want this extra constraint:

• it ensures perfect consistency between training and inference. Are you absolutely sure that all the fitted encoders/models/optimal_threshold you will store to mlflow manually are the one which goes with you code version? If the pipelines are bound to each other, we ensure consistency.
• if kedro-mlflow is "aware" that two pipelines are bound, it will create automatically the "mlflow model" linked to your inference pipeline each time you create kedro run --pipeline=training thanks to a hook

Hence, you do not have to create manually your Mlflow Model each time you have trained a new model on nex data / changed a parameter / fix a bug in your code: this is done automatically, so you can litteraly API-fy any training run that succeeded in one click.

I strongly suggest that you clone and run the kedro-mlflow-tutorial, run the training pipeline, and serve the automatically created API. It should help clarifying things. Do not hesitate to ask for clarification if it still not clear.

[Jarathael]

My Workflow

1. It's a bit different here. I have:
   i. a preprocessing pipeline (mainly computing some new features)
   ii. a splitting pipeline (to extract targets and split my data into train/valid/test)
   iii. a training pipeline with a train node including a sklearn pipeline with all my transformation components plus my model and which uses cross-validation, a threshold optimization node and an evaluation node
   Also I tend to run the preprocessing + split pipeline once (since there is not much change yet) and run several training pipelines to save some time since the preprocessing takes a bit of time. Not sure if this is good practice overall
2. My optimal threshold is not on the catalog. I was not sure how to make him fit there. So it is logged as a metric currently. Not the best way I guess
3. Yes.
4. Actually I have define my different "transformations" and my model in train.yml config and instantiate a whole sklearn pipeline (using ColumnTransformer) the mlflow artifact of the model correspond to the sklearn pipeline: the transformers + the model. Also I don't have an inference pipeline yet. The thing is, I was not going to do a kedro pipeline for that for several reasons:
   i. The prediction are done online so I want to avoid having the pipeline loading the model everytime it gets called. I guess it would lead to a bottleneck in some extent.
   ii. I thought kedro was not meant to go into production so I was not sure how to procede
   So my "inference pipeline" is more like: a subclass of mlflow.pyfunc.PythonModel with a predict method tying everything from the raw input to the prediction (basically it calls the preprocessing node, applies the model and give the prediction based on the threshold) and a node in my training pipeline that log that as an artifact.

I'm clearly not used to work on a big project from scratch and to production. Also it's my first one using kedro ! So my workflow might not be the best unfortunately.

Explanation of what kedro-mlflow does under the hood

I really should do that for sure to get a better grasp on that feature.

I guess my code would need some revamp to fit a bit more the best practices. I really feel like it's a bit clunky and messy :/

Thank you for taking the time to help me :)

[Galileo-Galilei]

You workflow trigger some warning flags in my head, but I need some clarifications to make things clear:

1. For the training pipeline:
   1. Do we agree that this preprocessing pipeline will be run by you during inference, not by the user calling your model? You do not reallistically expect your users to perform such transformations by themselves? (it seems to be ok given that you say in 4.ii that you want to have it part of your inference pipeline)
   2. For robust testing, it is not a good practice to separate training / test / validation at the same place in your pipeline. The train/test split can be done before training the ML model (roughly what your are doing here, or even a bit further in the pipeline) and is used to calculate metrics and evaluate its performance "at training time". The validation set must be created on your raw data, before any preprocessing of any kind. The idea here is that preprocessing is part of your process and can bring a lot of trouble (what will happen if one of your categorical variable has an extra level which does not exist in train/test during your preprocessing on the validation set? what happens if your new features contain N/A on your validation set (but not on training/test) because you forgot one condition?...). The point is that when you create feature ou make a lot of assumption on your data: for more robustness, you should keep your valdiation set apart, and not look at it before final validation.
   3. This sounds a very standard training pipeline. Running separately the preprocessing+splitting and the training to save time is quite normal while you are experimenting, but when you train your final model I strongly recommend that you train the end to end pipeline. You can add several tags to your nodes (say ["entire_training", "preprocessing_and_splitting"], ["entire_training", "training_only"]) to filter them out (you can run a pipeline with the --only-nodes-with-tags option or something like that). So when experimenting you can run the sub-pipeline you want ("preprocessing_and_splitting" or "training_only") and when you are done you can run the end to end pipeline ("entire_pipeline").
2. Mlflow metrics are intended to be "human readable" and to filter out the runs, but if you want to reuse this value in your final model (which is the case here), you should persit it locally (e.g. as pickle) and save it as artifact
3. Some remarks:
   • I don't really understand why you need to add an extra config file for this, but maybe it is for readibility?
   • It is a good thing that you are familiar with the ColumnTransformer, because kedro-mlflow does something very similar with your Kedro pipelines. The idea with ColumnTransformer (or sklearn.Pipeline) is to have a class with 2 methods: fit (=your training pipeline) and transform (=your inference pipeline) which compose estimators (=your nodes) which have themselves a fit and transform method (= you can use tags to specify whether a node should be included in the taining, inference or both). The main adavantage of using kedro is that you can run any arbitrary code (instead of being limited only to sklearn object with fit and transform method); the main advantage of using mlflow instead of storing all data in the attributes of an object (like ColumnTransformer) is that you can reuse the elements independently if needed (say if you need to reuse a label encoder without all the other data) and improve performance when loading them (e.g. if you have a very big artifact, like a big pre-trained model).
   • I really think that creating an inference pipeline is the first thing you should do right now: you must be able to predict on an end to end basis (including preprocessing) before you think about serving your model. Serving a model is just "technical" (i'm oversimplifying here, but from a data scientist point of view I think it's ok if you just see it as converting your pipeline into "something serviceable"), but you must provide the "functional" feature first, here your inference pipeline. After all, it is what your users care about, they don't care at all about your training pipeline :)
   1. You are right that globally kedro increase the overall overhead, but I don't think you should worry too much about this for several reasons:
      • unless you want to very strong constraints on speed defined by the type of project you are working for (say you are bidding advertisement location for a tech company on social networks where a few ms can change the price, or you are working for financial industry on high frequency trading), you should likely not prematurely optimise your inference speed IMHO.
      • if you use the kedro-mlflow class to convert your pipeline to an API (with pipeline_ml_factory, your model will be converted to KedroPipelineModel (which inherits from mlflow.pyfunc.PythonModel). When you serve this model as an API, all the artifacts are pre-loaded when the API is set up as Kedro MemoryDataSet, and they will not be reloaded for each prediction. This increases a lot inference speed.
   2. You are right that since kedro has not reached 1.0.0 yet, it is officially not recommended for production. If it can be of any help, I work in a huge bank which has a few dozens of kedro project in production, including some served as API which perform online inference for critical processes so they think it's good enough to go :)

Keep up learning Kedro, I really think it's worth the work :)

Thank you for taking the time to help me :)

You're welcome !

[Jarathael]

1. i. Yes. The idea is for the user to give not-preprocessed data to our API and then return a prediction to him.
   ii. I agree. I call that preprocessing but as I mentionned it mostly extract features from a SQL query and for some creates new features. The data from the query are URLs already broken into pieces and with DNS information. So for example I count the number of subdomains in the domain part. A more suitable word could be "data preparation" rather than "data preprocessing".
2. I'll do that then. Thank you !
3. • I don't have an extra config file. I just have a file for my "training pipeline like Kedro recommends: one param file for one pipeline.
   • So you think that I should use Kedro pipelines instead of ColumnTransformer ? I don't know if mentioned it but I'm using RandomizedSearchCV to hypertune all hyperparameters in my training node and this is why I used ColumnTransformer so all transformations are done after the data is split in the CV. I can't see how I can use those scikit-learn tools without putting all my processing and transformations in my model :/ Also I checked the kedro-mlflow-tutorial: what about processing features that relies on other data points like Imputer ? That's mostly why I put all "big" processing after splitting the data so I'm sure there is no leakage between the different set.
   • I'll work into that ! You are right, it's very important to have as best foundations as possible
     i. This is project comes within the scope of email filtering so the request frequency to the API will be quite high and it needs to be quite responsive, probably really much faster than 1s per request to the API. And it may be at most just a few predictions per request so not very batch oriented.
     ii. I see. I guess that's encouraging :)

Yes ! Kedro is a really good tool to structure ML project !

Side questions: I have several nodes using the same config (params:train) and then kedro-mlflow is logging those parameters multiple times. Does that show bad practice ? Or is this ok ?

[Galileo-Galilei]

Hi @Jarathael , did you manage to make it work as you want?

I plan to refactor the API for creating custom model to make it easier to use, if you want to beta test it, I'll keep you informed.

Regarding your questions:

So you think that I should use Kedro pipelines instead of ColumnTransformer?

Not really. Kedro pipelines offer more flexibility (e.g. if you want to perform data validation with hooks between nodes, or if you want to convert objects within nodes, e.g. between numpy and padnas for some reason) and help to review the code (which is visible within kedro-viz). On the other hand, ColumnTransformer enable to leverage the sklearn API for better performance (use numpy everywhere), perform GridSearch easily and avoid managing several artifacts manually. Both have advantages and drawbacks, and it is honestly not a great deal to choose one or the other. I accepth both when I do code review.

Also I checked the kedro-mlflow-tutorial: what about processing features that relies on other data points like Imputer ?

It's in this section, 3rd bullet point: "preprocessing node which produces an object fitted on data ". The key idea is your fit and transorm steps in 2 different nodes. The fit node return the imputer and is tagged ["training"], the transform takes the data and imputer as input and is tagged ["training", "inference"] so they can be used in the adequate pipeline.

This is project comes within the scope of email filtering so the request frequency to the API will be quite high and it needs to be quite responsive

• I have a mail classification project in production with kedro-mlflow which classifies thousands of emails per hour very easily, so it depends on how many emails/seconds you have, but it should be ok
• It is quite easy to scale mlflow which uses uvicorn as backend, you can predict on several cores in parallel

Side questions: I have several nodes using the same config (params:train) and then kedro-mlflow is logging those parameters multiple times. Does that show bad practice ? Or is this ok ?

I think it is more a bug of kedro-mlflow, I should rather check for duplication to avoid logging multiple times ^^' It is totally fine to reuse a parameter in several nodes in my opinion.

[Jarathael]

Hi @Galileo-Galilei !
I think I manage to do something correct. Basically I added a node to make the prediction, tagged all my nodes with training or inference and createe a "full pipeline" with pipeline_ml_factory.
Worth noting that I had some problems with the conda_env parameter since my version of mlflow was not the same as in the kedro-mlflow tutorial. I had to search in the mlflow source code to know how exactly everything worked under the hood.

It's in this section, 3rd bullet point: "preprocessing node which produces an object fitted on data ". The key idea is your fit and transorm steps in 2 different nodes. The fit node return the imputer and is tagged ["training"], the transform takes the data and imputer as input and is tagged ["training", "inference"] so they can be used in the adequate pipeline.

I see. The weird thing on my project is that there is a bunch of things done in only one node because the model is created, then a Random Search is done and the best model is fitted again. But I have something similar overall.

• I have a mail classification project in production with kedro-mlflow which classifies thousands of emails per hour very easily, so it depends on how many emails/seconds you have, but it should be ok
• It is quite easy to scale mlflow which uses uvicorn as backend, you can predict on several cores in parallel

I don't know the exact number yet. I still need to discuss that.
I made some local test with the API and with the time linux command. For one single input, it takes about 350ms and for 1000 inputs it's 1.8s. It will more likely be several few inputs rather than a big batch at a time. But you seem to say that mlflow can predict on several cores in parallel so it would make those 350ms paralellized and not that big of an issue even if we have to call the API thousands of times on single inputs.

I think it is more a bug of kedro-mlflow, I should rather check for duplication to avoid logging multiple times ^^' It is totally fine to reuse a parameter in several nodes in my opinion.
No problem !

[Galileo-Galilei]

I tagged a node as training only because it was computing some "fixed" data based on a file. It reduced drastically the API time response.

Yes it's always the hard part to decide how much computation you want to do during the API call, because obviously if the pipeline is slow the API will be too.

Regarding the deployment , I am not very familiar with docker but having a quick glance at the mlflow code, it seems this commands needs some implentation for the specific flavor. Since we are using a custom model, it is not very surprising that it does not work "automagically". I think that we ususally not use the mlflow models build-docker command but build the docker image "manually" when we deploy our kedro-mlflow models, do you confirm @takikadiri ?

[takikadiri]

Hi @Jarathael,
This is a common issue when using mlflow custom model. The issue occur because the environment that trained the model is not the same as the environment that will serve/run it. The custom model is intended to be portable.

Mlflow provide a way to declare the dependecies of the custom_model, by giving it a conda.yaml containing, conda and pypi dependecies/packages that were used in training time. The pyfunc log_model method provide also an entry to the project source code that the custom model (the pickle) will use to import your project's modules.

By using kedro-mlflow you can give a conda.yaml (as dict python) to the pipeline_ml_factory, but unfortunately we don't give yet an entry to point to your source code folder, we plan to add it very soon, see this issue for more details.

Now to get more in details about your specific error, can you share your traceback/Exception with us ? And can you clarify if you gived any dependencies/package to the pipeline_ml_factory ?

[Jarathael]

Hi @takikadiri,
Thank you for the reply !

I followed what is done in kedro-mlflow tutorial when calling pipeline_ml_factory. Basically here is my code for this:

full_pipeline = ep_pipeline + dp_pipeline + tts_pipeline + train_pipeline
inference_pipeline = full_pipeline.only_nodes_with_tags("inference")
training_pipeline_ml = pipeline_ml_factory(
    training=full_pipeline.only_nodes_with_tags("training"),
    inference=inference_pipeline,
    input_name="url_raw_features",
    model_name="urlclassifier",
    conda_env={
        "python": 3.6,
        "dependencies": {
            "pip": {"urlclassifier": PROJECT_VERSION},
        }  
    },
    model_signature="auto"
)

It's worth noting that I also tried to use mlflow models docker-build with the kedro-mlflow tutorial to be sure and so it would make things easier to test on your side:
mlflow models build-docker -m "runs:/<run-id>/kedro_mlflow_tutorial" -n "<image-name>"

And then:
docker run -p <port>:8080 "image-name"

The error in the console is pretty long to put it here but overall I have ModuleNotFoundError: No module named 'kedro_mlflow'
Also building the image with the previous command seems to use the default python installation of my computer, which is 3.9.5, instead of the current environment (I used 3.8 although the tutorial use 3.7 because I don't have this version on my work laptop)
And right after the module not found error I have this:

2021-09-15 13:32:40 +0000] [62] [INFO] Worker exiting (pid: 62)
Traceback (most recent call last):
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 209, in run
    self.sleep()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 357, in sleep
    ready = select.select([self.PIPE[0]], [], [], 1.0)
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
    self.reap_workers()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/miniconda/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 229, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 342, in halt
    self.stop()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 393, in stop
    time.sleep(0.1)
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
    self.reap_workers()
  File "/miniconda/lib/python3.9/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
Got sigterm signal, exiting.

I don't know if this error is linked to the module being not found.

The errors I get from my work project are pretty much the same.
I got quite some time before getting to real deployment though because I want to add some monitoring to get a highly functioning prototypes of the whole thing. But in the end, I guess that building a docker image out of the whole pipeline model mlflow API would make the deployment easy on any server.

[Galileo-Galilei]

Hi @Jarathael, did you manage to make it work? This issue in the tutorial is very likely the same issue you have :

• you have to make sure the conda.yml is complete and accurate (=it contains all the dependencies needed by your pipeline, e.g. import pandas)
• you have to make sure your own package is installed inside the image (=if you import from urlclassifier.pipelines.nodes import my_function, you must be sure that your own package urlclassifier is installed. We plan to provide better support for this in further releases

Notice that conda will try to install the packages from conda-forge or pip, so your package must be deployed somewhere it will be installable. So you can either:

• deploy your package to PyPI (or an internal repository manager if your enterprise has one): not recommended for public project, but recommended in an enterprise setup.
• create the image manually, and create the environment with everything you need inside, then run mlflow serve with the --no-conda flag to use the conda env you have created manually instead of creating it from the conda.yml of the mlflow model.

[Jarathael]

Hi @Galileo-Galilei,
I did not take the time to test it more since for now we only want a first proof of concept so I can set up things manually.
However if I understood correctly I just have to deploy my project to PyPI or the internal repository manager of my enterprise for mlflow to install it in the docker image.

I guess I can mark this discussion as answered.
EDIT: I have one more question link to this one though but I guess I should open a new discussion for it. Let's say I have a Kedro pipeline composed of different sub-pipelines (a vectorization pipeline and a model pipeline, just to simplify). I use kedro-mlflow to save it as one pipeline (vectorization + model) to be sure it corresponds to the same experiment etc. However, in production I would like to use those two pipelines separately. Is that something manageable ?

Thank you for your help :)