Dask

Reference: Dask Kubernetes Helm setup

The following Helm values file configures Dask workers with specific CPU/memory limits and installs additional Python packages on both workers and the Jupyter notebook server.

cat extra-config.yaml
worker:
  replicas: 4
  resources:
    limits:
      cpu: 1
      memory: 0.5G
    requests:
      cpu: 1
      memory: 0.5G
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: numba xarray -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: sklearn matplotlib s3fs dask-ml --upgrade

# We want to keep the same packages on the worker and jupyter environments
jupyter:
  enabled: true
  serviceType: NodePort
  env:
    - name: EXTRA_CONDA_PACKAGES
      value: numba xarray matplotlib -c conda-forge
    - name: EXTRA_PIP_PACKAGES
      value: dask_kubernetes s3fs dask-ml --upgrade

Install or upgrade the Dask Helm release using the custom values file defined above.

helm install k3sdask dask/dask -f extra-config.yaml
helm upgrade k3sdask dask/dask -f extra-config.yaml

Once Dask is deployed, you can connect to it from a Jupyter notebook and dynamically scale workers using the dask_kubernetes library.

```python from dask_kubernetes import KubeCluster cluster = KubeCluster.from_yaml(‘pod.yaml’) cluster.scale(1)