This document describes the guidelines for using PySpark 3.X through zep-pelin notebook on the EVOLVE dashboard. We provide a simple ETL example that loads a 2.5 GB dataset and performs an SQL query. Finally, we provide the configuration for enabling CPU only as well as GPU accelerated execution in PySpark 3.X.

For any issues or questions please contact aferikoglou@microlab.ntua.gr