A detailed guide on how Celery asynchronous task queue works. — Celery is a very popular and powerful asynchronous task queue. It supports both real-time operations and task scheduling. In one of my previous tutorials [1], I have demonstrated how to schedule a task using Celery. In this tutorial, I am going to demonstrate how to build an asynchronous task queue…

How to develop a more complicated web application to manage a project or task progress. — In the last post, I have demonstrated how to build a simple (static) web application using Flask AppBuilder (F.A.B). In this post, I am going to further demonstrate how to develop a more complicated web application, which manages project/task progress. We will include a PostgreSQL database into our application. …

A step-by-step guide on how to use FAB to build a simple web app. — Flask is a very popular (micro) framework for web application development. While one can build everything from scratch directly based on Flask and Jinja2, extensions built on top of Flask are available to make web app development more efficient and effective. Flask AppBuilder (F.A.B) is one of them. As an…

Schedule Jobs 2: Schedule recurring jobs/tasks using Celery. — In the last tutorial [1], I have demonstrated how to schedule jobs using Crontab. In this tutorial, I continue to demonstrate how to schedule recurring jobs/tasks using Celery. Overview Celery is a popular and powerful (open source) asynchronous task/job queue based on distributed message passing. It supports both scheduling and queuing…

Storing/Retrieving Objects into/from a relational database is very normal in many use cases. For example, in ETL pipelines, although data warehouse usually depends on RDBMS, NoSQL data sources are not rare. Especially for the staging database, sometimes, hierarchical data types, such as JSON, XML, YAML, etc, need to be stored…

Schedule Jobs 1: How to use Cron and Python scripts to acquire and upload earthquakes data to a staging database every day. — Scheduling regularly recurring tasks is a very common use case in many different situations, such as automatic database backup, automatic data acquisition, and processing in ETL pipeline, automated ML model retraining and deployment, and so on. Based on the requirement and the nature of the tasks, different tools can be…

In this tutorial, I am going to demonstrate how to deploy a toy static web server using Nginx and Docker container. First, we need docker installed on our working machine. There have been a lot of materials (documents, tutorials, etc) available online demonstrating how to install docker in different operating…

Flask blueprint is a great way to structure flask application by organizing views and other code into separate and reusable functionality components. It is conceptually similar to the applications in Django framework, and makes Flask applications easier to scale up. There has been a lot of tutorials online explaining how…

Apache Spark is a very powerful analytic engine for big data processing, and has become more and more popular in different areas of big data industry very recently. Spark includes built-in modules for streaming, SQL, machine learning and graph processing. It provides high-level APIs in Java, Scala, Python and R. …