Virtualized GPU Passthrough for Deep Learning on Proxmox VE

Question

Pulumi · Accepted Answer

To set up a virtualized GPU passthrough for deep learning on Proxmox VE using Pulumi, you would need to create a virtual machine that can be configured for GPU passthrough. Unfortunately, Pulumi does not have direct support for Proxmox VE as a cloud provider, which means there's no dedicated Pulumi provider to manage resources on Proxmox VE.

However, Pulumi does support major cloud providers like AWS, Azure, GCP, and on-premises resources using VMware vSphere, which allows you to configure virtual machines with GPU passthrough for deep learning purposes. Given that Proxmox VE isn't supported, I'll show you an example of how to create a virtual machine configured with a GPU using the `vsphere` provider as an alternative on-premises solution.

The Pulumi `vsphere` provider can manage resources on a VMware vSphere server, such as creating and configuring Virtual Machines (VMs), managing networks, and others. In this example, we'll provision a VM with attached GPU devices that can be used for deep learning.

Below is a Pulumi program in Python that illustrates how you can provision a VM with GPU passthrough using VMware vSphere. For the specifics of GPU passthrough and deep learning environment configuration (like CUDA drivers, deep learning frameworks etc.), you would need to configure the guest OS and install the necessary components manually or using another configuration management tool since Pulumi's scope is limited to infrastructure provisioning and management.

```python
import pulumi
import pulumi_vsphere as vsphere

# Initialize connection to vSphere server
vsphere_provider = vsphere.Provider("vsphere-provider",
    allow_unverified_ssl=True,   # If your vSphere does not use verified SSL
    user="vsphere-admin-username",
    password="vsphere-admin-password",
    vsphere_server="vsphere-server-url"
)

# Create a datacenter object
dc = vsphere.Datacenter("datacenter", 
    name="datacenter-name",
    opts=pulumi.ResourceOptions(provider=vsphere_provider)
)

# Reference to a datastore where the VM will be located
datastore = vsphere.Datastore("datastore", 
    name="datastore-name", 
    datacenter_id=dc.id,
    opts=pulumi.ResourceOptions(provider=vsphere_provider)
)

# Reference the network for the VM
network = vsphere.Network("network", 
    name="vm-network-name", 
    datacenter_id=dc.id,
    opts=pulumi.ResourceOptions(provider=vsphere_provider)
)

# Create a virtual machine
vm = vsphere.VirtualMachine("deep-learning-vm",
    name="deep-learning-vm-name",
    resource_pool_id=vsphere.ResourcePool.get("resource-pool", 
        dc.mo_id).id,
    datastore_id=datastore.id,
    num_cpus=4,
    memory=16384,
    guest_id=vsphere.VirtualMachineGuestId.LINUX_GUEST.value,  # A constant representing a generic Linux VM
    network_interfaces=[vsphere.VirtualMachineNetworkInterfaceArgs(
        network_id=network.id,
        adapter_type=vsphere.VirtualMachineNetworkInterfaceAdapterType.VMXNET3.value
    )],
    # Define the GPU passthrough device
    pci_device_ids=["gpu-device-id"],
    extra_config={
        "hypervisor.cpuid.v0": "FALSE",
        "mks.enable3d": "TRUE",
        "svga.vgaOnly": "FALSE",
        "vcpu.hotadd": "FALSE"
    },
    # Attach an ISO for OS installation
    cdroms=[vsphere.VirtualMachineCdromArgs(
        datastore_id=datastore.id,
        path="iso-images/ubuntu-20.04.iso"
    )],
    disks=[vsphere.VirtualMachineDiskArgs(
        label="disk0",
        size=50
    )],
    opts=pulumi.ResourceOptions(provider=vsphere_provider)
)

# Export the IP address of the VM after it's provisioned
pulumi.export("virtual_machine_ip", vm.default_ip_address)
```

This program does the following:

1. Sets up a Pulumi provider to connect to your vSphere server.
2. Creates references to existing resources on your vSphere server such as the datacenter, datastore, and network to be used by the VM.
3. Provisions a new virtual machine with specified CPU, memory resources, and an attached PCI device (assumed to be the GPU).
4. Configures additional VM options necessary for GPU passthrough.
5. Attaches an ISO for a Linux distribution (e.g., Ubuntu) to be used for OS installation.
6. Defines a single disk where the operating system and other files will be stored.
7. Exports the IP address of the virtual machine for external access.

While this script is an example starting point, GPU passthrough often requires additional BIOS/UEFI settings and host configuration, which aren't in the scope of Pulumi and must be configured manually or via other automation tools. Also, ensure the `pci_device_ids` property is set to the ID of the GPU you want to pass through to the virtual machine, which you'd obtain from your vSphere environment.

Remember to replace placeholders like `vsphere-admin-username`, `vsphere-admin-password`, `vsphere-server-url`, `datacenter-name`, `datastore-name`, `vm-network-name`, `gpu-device-id`, and the ISO path with the actual values from your environment.

For more reference documents on using the VSphere provider, please see the [Pulumi VSphere Provider Documentation](https://www.pulumi.com/registry/packages/vsphere/api-docs/).