Step-by-Step tutorial to build a minimal CI/CD pipeline for your Python project using Travis-CI | by Youness Mansar | Oct, 2020

[ad_1]

Constructing CI/CD pipelines are nice if you wish to save time on testing your python code in a number of environments earlier than publishing/deploying your packages routinely. It is usually a strategy to see bugs early and to make sure some consistency and reproducibility to your growth course of.

I just lately labored on a challenge that implements a comparatively current method to making use of a deep studying method to structured knowledge, particulars of the method might be discovered right here: Training Better Deep Learning Models for Structured Data using Semi-supervised Learning. I wished to arrange a CI/CD pipeline to do the next:

  • Automated testing of the code at each merge request.
  • Computing and displaying the check protection for the grasp department.
  • Automated deployment of the python package deal/wheel to PyPi if a construct on the staging department passes the exams.

To try this I used Github, Travis-CI, and Codecov, each of that are free for open-source tasks.

1) Logins

Step one is to log in to Travis-CI along with your Github account, then go to settings and activate the repository that you simply need to work on:

After which do the identical with Codecov:

And eventually to PyPI, from which you should generate an entry token by going to account setting:

2) Including the PyPI token to Travis-CI:

To automate publishing the package deal, you should add the PyPI token to Travis-CI as an surroundings variable. In settings:

3) Code

The code must have a setup.py file in addition to a necessities.txt (if wanted). For instance, my code depends on a number of libraries like Tensorflow or Pandas, so I would like a requirement file like this:

pandas==1.0.4
numpy
==1.17.3
scipy
==1.4.1
matplotlib
==3.1.1
tensorflow_gpu
==2.0.1
tqdm
==4.36.1
scikit_learn
==0.23.2
tensorflow
==2.3.0

You’ll additionally must implement some exams and put them in a exams/ folder. An instance of a check in my code is to run a small coaching on an artificial coaching set and verify that the community discovered by operating an analysis on the check set:

from deeptabular.deeptabular import (
DeepTabularClassifier,
)
import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.metrics import accuracy_score

def test_build_classifier():
classifier = DeepTabularClassifier(
cat_cols=["C1", "C2"], num_cols=["N1", "N2"], n_targets=1, num_layers=1
)
df = pd.DataFrame(
{
"C1": np.random.randint(0, 10, dimension=5000),
"C2": np.random.randint(0, 10, dimension=5000),
"N1": np.random.uniform(-1, 1, dimension=5000),
"N2": np.random.uniform(-1, 1, dimension=5000),
"goal": np.random.uniform(-1, 1, dimension=5000),
}
)
df["target"] = df.apply(
lambda x: 1 if (x["C1"] == 4 and x["N1"] < 0.5) else 0, axis=1
)

check = pd.DataFrame(
{
"C1": np.random.randint(0, 10, dimension=5000),
"C2": np.random.randint(0, 10, dimension=5000),
"N1": np.random.uniform(-1, 1, dimension=5000),
"N2": np.random.uniform(-1, 1, dimension=5000),
"goal": np.random.uniform(-1, 1, dimension=5000),
}
)
check["target"] = check.apply(
lambda x: 1 if (x["C1"] == 4 and x["N1"] < 0.5) else 0, axis=1
)

classifier.match(df, target_col="goal", epochs=100, save_path=None)

pred = classifier.predict(check)

acc = accuracy_score(check["target"], pred)

assert isinstance(classifier.mannequin, tf.keras.fashions.Mannequin)
assert acc > 0.9

4) The pipeline

The pipeline utilized in Travis-CI is written as a YAML file. For instance, the one used within the deeptabular repository is:

language: python
python:
- "3.6"
- "3.7"
set up
:
- pip set up -r necessities.txt
- pip set up codecov
- pip set up pytest-cov
- pip set up .
script:
- pytest --cov-report=xml --cov=deeptabular exams/

after_success:
- codecov

deploy:
supplier: pypi
person: __token__
password: $TEST_PYPI_TOKEN
distributions: "sdist bdist_wheel"
skip_existing
: true
on:
department: staging

Subsequent select the python model to make use of:

python:
- "3.6"
- "3.7"

Then set up the necessities of the library plus the library itself, pytest and Codecov used for testing:

set up:
- pip set up -r necessities.txt
- pip set up codecov
- pip set up pytest-cov
- pip set up .

Run the exams and write the check protection outcomes as an XML:

script:
- pytest --cov-report=xml --cov=deeptabular exams/

Push the protection report back to codecov:

after_success:
- codecov

Lastly, publish the package deal as zip and wheel to PyPI:

deploy:
supplier: pypi
person: __token__
password: $TEST_PYPI_TOKEN
distributions: "sdist bdist_wheel"
skip_existing
: true
on:
department: staging

The package deal is then pushed to PyPI :

And the check protection outcomes can be found in Codecov:

Conclusion:

That is it, this pipeline runs the exams every time there may be code pushed to any department, and publishes the package deal to PyPI if the staging department is modified.

References:

  1. https://dev.to/oscarmcm/distributing-pypi-packages-using-api-tokens-in-travisci-1n9i
  2. https://docs.travis-ci.com/user/languages/python/

Code:

https://github.com/CVxTz/DeepTabular

[ad_2]

Source link

Write a comment