GitopsCentral

by

In this blog post, I will show you how to install apache airflow on ubuntu,

Introduction

According to Apache’s official web site, Apache airflow is a platform for programmatically author schedule and monitor workflows. It is one of the best workflow management system.

Airflow was originally developed by Airbnb (Airbnb Engineering) to manage their data based operations. Now, Airbnb open sourced it under apache license and hence got more popular.

Apache airflow makes your work flow little bit simple and organized by allowing you to divide it into small independent (not always) task units, So that it’s easy to organize and easy to schedule ones.

Your entire workflow can be converted into a DAG (Directed acyclic graph) with Airflow. Defining workflow makes your code more maintainable

Prerequisites

You need to have following thing installed already in order to install apache airflow, we will install this using pip (python interpreter)

  • python & pip
  • virtualenv
  • python-dev
  • libsasl2-dev
  • gcc
  • libffi-dev
  • libkrb5-dev

Installation

Let’s start by installing Prerequisites first, run below commands.

sudo apt-get install python-dev libsasl2-dev gcc
sudo apt-get install libffi-dev
sudo apt-get install libkrb5-dev
sudo apt install virtualenv

We are using virtualenv for this installation because we have issues with direct install on pip so let’s create a a virtualenv. So, let’s install virtualenv and configure it for the airflow

sudo virtualenv airflow_virtualenv
cd airflow_virtualenv/bin
source activate
export SLUGIFY_USES_TEXT_UNIDECODE=yes
export AIRFLOW_HOME=~/airflow
pip install apache-airflow
pip install apache-airflow[all]
airflow initdb

Finally, this should install airflow and you can start using it, also remember that you can export AIRFLOW_HOME in you bashrc profile.

Introduction

Let’s run the server and see if we can load the web page, execute the command below,

airflow webserver -p 8080

This will take about a few minutes for service to be up and running, Let’s test it on the browser. Finally, there you see the admin page!

Please check more tags on goacademy