[Prefect] Installation & Run

참고 : https://docs.prefect.io/latest/tutorials/

 

Overview

• Prefect는 데이터 파이프라인을 정의하고 관리하는 플랫폼입니다.
  • Prefect와 비슷한 Workflow 오픈소스로는 Apache Airflow가 있습니다.
• Flow를 스케쥴링 할 수 있고 스케쥴링 된 Task에 대한 상태를 확인 할 수 있습니다.
• Prefect는 Python 기반으로 코드를 작성합니다.
• Prefect는 크게 UI 와 Agent 또는 Worker로 나뉠 수 있습니다.
• Agent 또는 Worker는 Process, Docker와 Kubernetes 등등의 위에서 동작 할 수 있습니다.

 

Prerequisite

• Python 3.9
• Rocky Linux 8.7

1. Install Python 3.9 & Prefect

> yum -y update
> yum -y install python39 python39-pip python39-devel
> python3
Python 3.9.13 (main, Nov 16 2022, 15:31:39)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

> pip3 install prefect
> prefect version
 prefect version
Version:             2.10.5
API version:         0.8.4
Python version:      3.9.13
Git commit:          d48dfbbc
Built:               Thu, Apr 20, 2023 4:06 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

 

2. Basic Flow & Task 코드 작성

> vim main_flow.py

import sys
import requests
from prefect import flow, task

@task
def request_url_task(url):
    return requests.get(url).json()

@task
def print_task(result):
    print(result)
    
@flow
def main_flow(url: str):
    result_json = request_url_task(url)
    print_task(result_json)

# if __name__ == '__main__':
main_flow(sys.argv[1])

 

3. Flow Run

> python main_flow.py "http://time.jsontest.com/"
15:34:16.285 | INFO    | prefect.engine - Created flow run 'thoughtful-falcon' for flow 'main-flow'
15:34:16.579 | INFO    | Flow run 'thoughtful-falcon' - Created task run 'request_url_task-0' for task 'request_url_task'
15:34:16.579 | INFO    | Flow run 'thoughtful-falcon' - Executing 'request_url_task-0' immediately...
15:34:17.247 | INFO    | Task run 'request_url_task-0' - Finished in state Completed()
15:34:17.276 | INFO    | Flow run 'thoughtful-falcon' - Created task run 'print_task-0' for task 'print_task'
15:34:17.277 | INFO    | Flow run 'thoughtful-falcon' - Executing 'print_task-0' immediately...
{'date': '04-29-2023', 'milliseconds_since_epoch': 1682750057002, 'time': '06:34:17 AM'}
15:34:17.379 | INFO    | Task run 'print_task-0' - Finished in state Completed()
15:34:17.415 | INFO    | Flow run 'thoughtful-falcon' - Finished in state Completed('All states completed.')