How to Create ETL in Luigi

Luigi — Kapernikov

The Almighty ETL

  • Enhanced business intelligence
  • Timely access data
  • Enhance quality and consistent
  • High return on investment (ROI)

What is Luigi?

  • Task is about doing something as the basic data processing step in a pipeline. For instance, a task can implement order processing or report generation
  • Target is doing after the task has finished which is the second building in the pipeline. For instance, the report generation create a CSV file with the actual report
  • Requires contains all the tasks instance that must be executed before the current task.
  • Output is the step to store the output task. This output contains one or more target objects.
  • Run is the step to do several logics to run the ETL
Pipeline in Luigi

Terminal

  1. writing python3 -m venv ~/Documents/demoday, this is my folder path (~/Documents/demoday)
  2. Activate the virtual environment by writing this source ~/Documents/demoday/bin/activate
  3. Install Luigi by writing this pip install luigi
  4. Check the tools in your folder by writing pip list. Here is the result
Luigi has been installed

Pycharm

The steps to create a python project to run Luigi

Hands-on ETL using Luigi

1. Practice creating a report of sales

2. Practice compiling the existing file into one file

The pipeline of your ETL

Conclusion

--

--

--

Love in data and investment | Senior Data Analyst | www.linkedin.com/in/ariprabowo/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

[CC Lab 21春] Mixed Reality Interface for AI-based Musical Expression vol.1

Validating XML documents with XSD

Build a Weather Dashboard that sends alerts using a Weather Data API

Weather Data API Weather Dashboard

How I Stayed Focused While Learning To Code

Production Code and Data Scientists

BUILD YOUR FITNESS APP — STEP BY STEP GUIDE

Genaro Network: What are the Advantages of Distributed Storage?

The Perfect Dumpling Quest

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ari Sulistiyo Prabowo

Ari Sulistiyo Prabowo

Love in data and investment | Senior Data Analyst | www.linkedin.com/in/ariprabowo/

More from Medium

Installing SQL Server and Microsoft SQL Server Management Studio on Windows Server 2019 VM.

Encrypting Secret Data Stored in ETCD Database.

Push large data files onto Git Repository

CI/CD pipeline using Github action and Docker hub