How to Install AMD FPGA Tools in x86 Ubuntu VM on M2 Mac

If you've been following me for a while, you're probably aware of my preference for Apple/Mac OS since I worked at an Apple Store as a technician that was near my university campus throughout my undergrad. Since I was working there when iCloud was originally rolled out (ouch, I'm remembering my age) I was forced to get to know how it worked with Mac OS inside & out. This left me with minimal energy to get to know another OS as intimately.

Joke's on me, because my interest in FPGA/digital design led me straight into the belly of the beast that is Linux. However, since I has well & truly been sucked into the ecosystem of iCloud, my solution was to run Linux in a virtual machine so I could use the FPGA development tools I needed while still being able to access my files on the Mac side. This also get me a way to back everything up to iCloud to save me from my own stupidity during the times I thought I knew what I was doing in Linux, but actually just set off of nuclear bomb and destroyed everything.

On the Intel-based Macs, I had created a very stable VM setup using the pro version Parallels. And when Apple came out with their custom ARM-based silicon with the M1/M2 chips, I was initially excited. Until I realized that some of the key FPGA tools I use on the Linux side absolutely do not run on anything but AMD64 architecture.

For the past couple of years, I've been fighting with getting a usable VM setup on an Apple silicon Mac (and praying nothing happens to my last generation Intel-based Mac). While Parallels has been rolling out emulation support for running x86-64 Docker containers and binaries in ARM-based Linux virtual machines, this absolutely does not work with the AMD FPGA tools for any version of Vivado, Vitis/XSDK, or PetaLinux. No matter what I did to hack around it, the AMD FPGA tools need a x86-64 environment (even putting Vivado into its own x86-64 Docker container on an ARM Linux VM crashed and burned).

With the release of the Apple silicon Macs, UTM popped up as a new virtual machine host specifically for Apple silicon. UTM utilizes Apple's Hypervisor virtualization framework to run ARM64 operating systems on Apple Silicon at "near native speeds" (which I can vouch for from personal experience so far). But it unique from other virtual machine hosts at the moment, as it is the only one so far that supports full emulation of other architectures.

The first several versions of emulated x86-64 (AMD64) Ubuntu VMs I created were too laggy/unstable to be anywhere near usable for daily use. But as UTM has updated over the past couple of years, I finally was able to get an x86-64 (AMD64) Ubuntu 22.04 LTS VM set up on my M2 MacBook Air with a usable install of Vivado, Vitis, and PetaLinux version 2023.2. While it is a bit laggy and I do experience some odd crashes here and there, it's certainly a usable solution for the time being.

Install UTM

While you can build UTM from source for free because it is a fully free and open source project, I chose to purchase it from the App Store because I truly believe it is a great project that deserves my support:

I'm using what is currently the latest version of UTM, which is version 4.4.4 on Mac OS Sonoma 14.2.1. It is important to use at least this version when following this tutorial because a lot of major updates have happened with UTM recently that made this setup usable.

Download AMD64 Ubuntu ISO

Download the iso image file for Ubuntu from Canonical's website here. It is important that you download the AMD64 ISO for Ubuntu Desktop.

The AMD64 ISO was the default version that downloaded for my on my M2 Mac, but if a different version downloads for you, just scroll down the webpage a bit to the link for "our alternative downloads" which will take you to a list of the various Ubuntu versions to download.

Create Emulated VM in UTM

Upon launching UTM, select the option to Create a New Virtual Machine:

On the first page of the setup windows, select the option to Emulate:

Select Linux:

Then under Path for Boot ISO Image, browse to the download of the AMD64 Ubuntu Desktop ISO

For the initial setup, I left all of the default selections for the hardware of the emulated x86-64 architecture:

For the size of the hard drive of the VM, I chose 512 GBs since I was only planning to install the one version of the AMD tools and this was my test VM. If you plan to install multiple versions however, I recommend making the allocated hard drive size for the VM at least 1TB.

I skipped the shared directory option for now as I am still figuring out how I want to set this up. But this will ultimately be how I back up my source files in Ubuntu to iCloud again.

On the Summary page, give the VM the desired name and check the option to Open VM Settings before clicking Save:

The VM settings window will open. I like the option in the Information tab to leave notes. I used this quite a bit when I was testing different VMs while ultimately trying to figure out the right settings in the VM host and inside Ubuntu in order to install/run the AMD FPGA tools.

Under the System tab, increase the amount of RAM the VM can use to be all but a few GBs of what your system has. I found this has one of the biggest impacts on system stability within the VM. You might have to tweak this a bit on your own machine because allocating too much also made the VM just as unstable as allocating too little memory. I have 24GB of RAM in my MacBook, and I found that the sweet spot for an emulated Ubuntu VM was 14GB.

I also allocated 6 out of the 8 cores my MacBook has which was a big factor in the VM stability as well. Enabling the option to Force Multicore was also a must:

QEMU is ultimately what's running under the hood to support the emulation of the x86-64 architecture. For tweaks of QEMU, I found that UEFI Boot, RNG Device, and Balloon Device were all necessary:

A recent upgrade of UTM and key requirement of mine is the option to connect to USB devices on the host machine:

Another super handy feature that is more helpful than one might realize is the ability to share the clipboard between the host and VM. This has no impact on the performance of the emulated VM, but I highly recommend it just for day-to-day use.

The emulated display card is another huge factor for overall VM performance and stability. This was another setting I had to play with for a while to test different options. I ultimately found that the virtio-ramfb-gl (GPU Supported) option was the best with Upscaling set to Nearest Neighbor and Downscaling set to Linear:

I left all of the default options for the Network settings:

As well as for Sound settings:

After updating all of the desired settings for the emulated VM, click Save.

Install Ubuntu

With the VM set up, the next step is to fire it up and install Ubuntu from the ISO image pointed to in the set up screens:

The behavior will be the same as a fresh install of Ubuntu on a new hard drive. Just follow the prompts for the Install Ubuntu option in the Welcome screen:

Optimize Ubuntu Environment for Emulation on Mac

I started with my usual environment prep for installing Vivado/Vitis/PetaLinux that I've documented in my previous install tutorial, but I found there were a few extra packages to install and system settings to tweak in Ubuntu so it would be more stable on the emulated hardware in QEMU.

The first thing I found was a must (since I'm using Ubuntu 22.04) was using the Xorg display server versus Wayland as the display manager. Wayland didn't seem to know what to do with the emulated hardware so things were very laggy and had trouble rendering. Firefox in particular couldn't load any webpages without crashing using Wayland. The underlying instance of VS Code in Vitis also had issues rendering with Wayland.

Next, change the shell from dash to bash, add the 32-bit architecture that Vivado and PetaLinux, and enable the userspace out-of-memory killer (systemd-oomd.service). The OOMD service is a necessity in the emulated environment so unused processes can be killed before an out-of-memory event occurs, throwing the whole system into a kernel panic (read more about it here).

Note: the 32-bit architecture was the roadblock that I could not overcome to get the AMD Tools working in an ARM64 version of Linux with emulation support for x86 containers. I couldn't add both the AMD64 and I386 architecture at the same time. If any Linux Jedi can explain how to do this in the comments, I would love to know!

~$ sudo dpkg-reconfigure dash
~$ sudo dpkg --add-architecture i386
~$ sudo systemctl enable systemd-oomd.service
~$ sudo apt update 
~$ sudo apt upgrade

The main issue I found with the emulated environment were pretty much all graphics drivers related. Which is fair, because at this time of writing, better graphics support for emulation is one of the big things UTM still has in development.

The following extra packages are various graphics and graphics adjacent dependencies that I found were necessary for emulated x86 Ubuntu through copious amounts of trial & error:

~$ sudo apt install ffmpeg mesa-utils openjdk-11-jre-headless
~$ sudo apt install nodejs npm
~$ sudo apt install libnss-nis libnss-nisplus
~$ sudo apt install software-properties-common apt-transport-https wget gpg dirmngr

Even though I had installed the Mesa driver utilities, I noticed that in looking in the information about the OpenGL/GLX implementation running on the X display weren't actually utilizing a proper/stable version of the Mesa drivers:

~$ glxinfo | grep OpenGL
OpenGL version string: 2.1 Mesa 23.0.4-0ubuntu1~22.04.1

So I installed them manually:

~$ sudo add-apt-repository ppa:kisak/kisak-mesa -y
~$ sudo apt update 
~$ sudo apt upgrade

And validated they were indeed the driver the OpenGL/GLX implementation was now using:

~$ glxinfo | grep OpenGL
OpenGL version string: 2.1 Mesa 23.3.3 - kisak-mesa PPA

Then I was able to install the rest of the Vivado/Vitis/PetaLinux dependencies as normal:

~$ sudo apt-get install gparted xinetd gawk gcc net-tools ncurses-dev openssl libssl-dev flex bison xterm autoconf libtool texinfo zlib1g-dev
~$ sudo apt-get install iproute2 make libncurses5-dev tftpd libselinux1 diffstat chrpath socat tar unzip gzip tofrodos lsb libftdi1 libftdi1-2
~$ sudo apt-get install lib32stdc++6 libgtk2.0-0:i386 libfontconfig1:i386 libx11-6:i386 libxext6:i386 libxrender1:i386 libsm6:i386 tree openssh-server
~$ sudo apt-get install debianutils iputils-ping libegl1-mesa libsdl1.2-dev pylint python3 cpio tftpd-hpa gnupg zlib1g:i386 haveged perl xvfb
~$ sudo apt-get install gcc-multilib build-essential automake screen putty pax g++ python3-pip xz-utils python3-git python3-jinja2 python3-pexpect
~$ sudo apt-get install liberror-perl mtd-utils xtrans-dev libxcb-randr0-dev libxcb-xtest0-dev libxcb-xinerama0-dev libxcb-shape0-dev libxcb-xkb-dev
~$ sudo apt-get install util-linux sysvinit-utils cython3 google-perftools patch diffutils ocl-icd-libopencl1 opencl-headers ocl-icd-opencl-dev
~$ sudo apt-get install libncurses5 libncurses5-dev libncursesw5:amd64 libncursesw5-dev libncurses5:i386 libtinfo5 libstdc++6:i386 libgtk2.0-0:i386 dpkg-dev:i386

And add my user to the dial out group for connecting to USB devices:

~$ sudo adduser $USER dialout

Add the TFTP server for PetaLinux:

~$ sudo gedit /etc/xinetd.d/tftp
~$ sudo mkdir /tftpboot
~$ sudo chmod -R 777 /tftpboot
~$ sudo chown -R nobody /tftpboot
~$ sudo /etc/init.d/xinetd stop
~$ sudo /etc/init.d/xinetd start

Then update and upgrade everything for good measure:

~$ sudo apt update 
~$ sudo apt upgrade

Install AMD FPGA Tools

With the environment finally up and running in a stable enough manner, I was able to run the AMD FPGA installer. I initially tried to use the web installer since I only had the hard drive for the Ubuntu VM 512GB, but it absolutely refused to work. It would get to 99% of the initial download then error out with a vague network error. I eventually gave up and just used the single-file download (SFD) installer.

I found it odd that the SFD installer was able to download through Firefox the first try with no errors, so I really don't understand what the web installer was crashing & burning on.

I ran the xsetup script from the SFD directory:

~$ cd ./Downloads/FPGAs_AdaptiveSoCs_Unified_2023.2_1013_2256/
~/Downloads/FPGAs_AdaptiveSoCs_Unified_2023.2_1013_2256$ sudo ./xsetup

It did crash the first time, but ran the second time successfully:

And I was able to run the script for checking for missing libraries and the cable drivers as usually with no issue:

~/Downloads/FPGAs_AdaptiveSoCs_Unified_2023.2_1013_2256$ cd /tools/Xilinx/Vitis/2023.2/scripts/
/tools/Xilinx/Vitis/2023.2/scripts$ sudo ./installLibs.sh
/tools/Xilinx/Vitis/2023.2/scripts$ cd /tools/Xilinx/Vivado/2023.2/data/xicom/cable_drivers/lin64/install_script/install_drivers/
/tools/Xilinx/Vivado/2023.2/data/xicom/cable_drivers/lin64/install_script/install_drivers$ sudo ./install_drivers

After downloading the PetaLinux installer from AMD's website, I was able to install it as usual with no issue as well:

~$ sudo mkdir -p /tools/Xilinx/PetaLinux/2023.2/
~$ sudo chmod -R 755 /tools/Xilinx/PetaLinux/2023.2/
~$ sudo chmod 777 ./Downloads/petalinux-v2023.2-10121855-installer.run
~$ sudo chown -R whitney:whitney /tools/Xilinx/PetaLinux/2023.2/
~$ ./Downloads/petalinux-v2023.2-10121855-installer.run --dir /tools/Xilinx/PetaLinux/2023.2/

And finally the Xilinx Runtime Library for Ubuntu 22.04:

~$ cd ./Downloads/
~/Downloads$ sudo apt install ./xrt_202320.2.16.204_20.04-amd64-xrt.deb

I did have to apply the patch of disabling the GPU acceleration when launching the underlying VS Code application of Vitis Unified that I outlined in my other project post here.

I validated the install by recreating my recent tutorial for creating/running the vector addition accelerated application on the Kria KD240 since that would exercise Vivado, Vitis, PetaLinux, and XRT all in one project:

Things were definitely slower than my virtualized Ubuntu VM on my Intel-based Mac, but not unusable.

Connecting USB Devices through UTM

One key thing I discovered in the UTM forum after I had already nuked an external hard drive is that storage devices such as external hard drives and USB drives need to go through USB hub. I found that all devices seemed to work better through a USB hub for the current version of UTM (including the JTAG/UART connection to the KD240).

Conclusions

So this setup isn't perfect and is still prone crashes and unique problems. However, it's still usable like I've mentioned.

It's also worth noting that AMD does not offer any sort of support for running Vivado/Vitis/PetaLinux in virtual machines to begin with so there definitely isn't any official support for running them in an emulated VM.

I'm sure some minor update in Ubuntu will spectacularly break everything so I will update as I find them. Wish me luck!