How to install Airflow on local machine

Airflow 2.0 is the hot trend, it’s the go to solution for pipeline development or orchestration in the current digital transformation era. It’s an open source platform developed by community to programmatically author, schedule and monitor workflows.

Astronomer

“The Enterprise Framework for Apache Airflow”

Astronomer platform is completely developed over apache airflow and is used by many fortune 500 organisation in there digital transformation journey as it provides good support to various public cloud.

Best way to learn airflow is by doing some coding. Fail fast and learn sooner !

Pre-requisite —

1. Ubuntu — https://ubuntu.com/desktop/features

Ubuntu on windows allows you to use Linux terminal CLI utilities such as bash, ssh, git, apt and many more.

Download the ubuntu software from above mentioned link and setup the profile.

While setting up profile one will be prompted to set username and password.(Please refer below shared snapshot for your help)

Ubuntu profile created successfully

2. Docker

Astronomer CLI installation required docker(V18.09 or higher). Run below commands to install docker on ubuntu.

Install using the repository

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Add Docker’s official GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Use the following command to setup the stable repository

echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Confirm docker is installed

docker version

Note: If you see below warning

Check if docker service is available.

sudo service --status-all

Use below code to check if docker is running —
sudo service docker status

Docker not running

To start docker service, run below command :

sudo service docker start

Check status again

sudo service docker status

Docker Active

Install Astronomer CLI

Execute below commands on ubuntu to install the astronomer on the local machine

Install the cURL

curl -sSL https://install.astronomer.io | sudo bash

Astro installed successfully

Confirm if above command executed successfully

astro version

Astro Version

Initialize an Airflow project

Create a new directory

mkdir <directory-name> && cd <directory-name>

New Directory

Create necessary project files

astro dev init

Project files are created successfully.

We are almost done, let’s start the the airflow —

astro dev start

Note:

Access issue on the docker directory (unix /var/run/docker.sock: connect: permission denied)

Run below command to resolve it —

sudo chmod 666 /var/run/docker.sock

Required docker images are getting downloaded

Astronomer Started :)

Access the Airflow UI from your local Airflow project. To do so, go to http://localhost:8080/ and enter username and password both as admin

Login Window

Airflow GUI

And you are all set to explore Airflow :)