Skip to content

PREFECTv3.md

Latest commit

 

History

History
80 lines (53 loc) · 3.52 KB

PREFECTv3.md

File metadata and controls

80 lines (53 loc) · 3.52 KB
Raw

Prefect v3 Notes

If there is not a prefect server setup locally, prefect will start one automatically for your run. This can cause some issues with flow runs not finishing correctly if an Exception is encountered.

Recommended usage is to launch the prefect server as a background task and configure your account point to the local server.

Prefect now runs as a system process

This is setup in the head-node deployment scripts.

To see the status:

sudo systemctl status prefect-server

By default, the server runs on the localhost 127.0.0.1 port:4200

Configure your profile to use the running server

prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"

Additional settings to persist results are already set in the systemd service

prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT="true" prefect config set PREFECT_TASKS_DEFAULT_PERSIST_RESULT="true"

These settings might need to be added to user's ~/.prefect/profiles.toml or run the prefect config set steps above

export PREFECT_API_URL="http://localhost:4200/api"
export PREFECT_RESULTS_PERSIST_BY_DEFAULT = "true"
export PREFECT_TASKS_DEFAULT_PERSIST_RESULT = "true"

You can connect to the Prefect UI by using an ssh tunnel and connect to the webserver on your machine.

Connect via SSH with a tunnel from 4200 to 4200 ssh -L local_port:destination_address:remote_port user@remote_server_addres

Example: ssh -i ~/.ssh/id_rsa -L 4200:localhost:4200 username@sandbox

Open a web browser and open the URL: http://localhost:4200/dashboard

Stop the prefect server if necessary

prefect server stop

Stopping the prefect server may be required for debugging issues with running cloudflow on new head nodes.

There are also useful Prefect CLI tools:

  • prefect --help
  • prefect flow-runs ls

To cancel a currently running flow:

Use prefect flow-run cancel flowrun-id. The id can be obtained from "prefect flow-runs ls" command. The Name was output at the beginning of the run and can be found in your log output.

Example output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃                                   ID ┃ Flow          ┃ Name           ┃ State     ┃ When         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ 0c7df0f5-c2f4-425c-ae73-bfd53ca8ec50 │ hindcast-flow │ offbeat-impala │ RUNNING   │  1 hour ago  │
│ b66329bd-4e55-4ff5-a442-5ac2fffca3e6 │ hindcast-flow │ fuzzy-bug      │ COMPLETED │ 17 hours ago │
│ d92a59f0-7ac8-4b4f-9616-caade4a076cf │ hindcast-flow │ crafty-potoo   │ COMPLETED │ 20 hours ago │
│ 199e77cb-99ed-4b12-8b8b-15f88d091399 │ hindcast-flow │ perky-hawk     │ COMPLETED │ 21 hours ago │
└──────────────────────────────────────┴───────────────┴────────────────┴───────────┴──────────────┘

Example: prefect flow-run cancel 'a55a4804-9e3c-4042-8b59-b3b6b7618736'