RF Data Converter Loopback Tutorial

Table of Contents

This tutorial shows how to build a project from scratch for the ZYNQ RFSoC that does the following:

1. Streams an arbitrary complex waveform using DMA from a Python-created array into the FPGA, going into a dual-clock FIFO
2. The FIFO feeds the RF Converter DAC input for Tile 228 DAC 0 which connects to the DAC_B SMA output
3. Inside the RF Converter, mix the waveform with a carrier wave at some frequency $\omega_0$ that you specify in Python,
4. Outputs a real analog waveform on the DAC_B SMA output using a sampling frequency $f_s=6.88128$ GSps that is specified when the FPGA project is built,
5. You connect the DAC_B output to the ADC_A input
6. Converts the incoming signal using a different sampling frequency $f_s=4.9152$ GSps,
7. Mixes it with the a carrier frequency that you can set in Python
8. Streams the captured complex data back into memory using DMA for analysis etc.

All is controlled from Python with PYNQ, and all filters will be set in the FPGA project to keep things in the 1st Nyquist zone.

The files needed to recreate everything are here:

There are several complications that need to be discussed in order to make such a project work, notably 1) how to make sure that when we read data from the ADC in Python, we see the signal that we sent out (also in Python); 2) how to do this using the AXI DMA engines. The former is an issue of synchronization, and the latter involves details about AXI DMA, a powerful yet also constrained way to do direct memory access.

First, the synchronization issue. Imagine some python code where you first put some data into a large array, send it via DMA to the DAC, route the data from the DAC SMA output into the ADC SMA input, DMA read it back into an array, and then do something in Python (Fourier analyze, plot, etc). First of all, the DAC in the RF Converter runs continuously: data in, converted to analog, voltage out. There's no "start" or "stop" signal. So if you just send a burst of data to the DAC in the RF Converter, the DAC will immediately process this data and an analog voltage will appear on the SMA output connector. If you run the cable from this connector into the ADC, it will do the same thing the DAC does only in reverse: analog in, conversion, digital data out. No "start" or "stop", continuous operation. But in Python, you want to make sure that when you read the data from the ADC, you are seeing the waveform that you sent to the DAC. If you just DMA an array from DDR to the DAC and then read the ADC, you probably will see only noise unless you control the timing in some way.

The way we will solve this problem is simple: we define an array in DDR memory, and then tell the hardware to DMA this memory to a FIFO (and we always use a FIFO to cross clock boundaries) in a mode where when it gets to the last word in the memory and all the data is transferred, it immediately starts back at the beginning and does it again, in cyclic mode. This way as long as we read the same amount of data (or less) than we sent, we will always see the waveform.

There are 2 sub problems with this that we need to take care of. The first involves the waveform we are sending (waveform wrapping), and the second involves continuity in reading the FIFO that contains the ADC data (FIFO continuity).

and use that in equation $\ref{mint}$ to modify $f$ slightly so that that the ratio of frequencies is a rational number to get $f'$: $$f' = \frac{m' f_s}{N}\nonumber$$ As a concrete example, below we have constructed a project that has a TX_FIFO read clock of 172.032 MHz. We want to generate a tone that has a $f=100$ MHz frequency, and we will send an array that has $2^{24}=16.7M$ points. If we use those to calculate $m$ we would get $m=f\cdot N/f_s=100M\cdot 2^{24}/172.032M=9752380.95$. Turning that into an integer gives $m'=9752389$, and that gives us a value for $f'=99.99999023$, which is different from $f$ by $\sim 9.8Hz$.

If your waveform is more complex, then you have to work a little harder, but not much. Say your waveform is generated from a spectrum of frequencies $f_i$, and you want to inverse Fourier that spectrum and put it into a buffer of size $N$ that is from a sampling with frequency $f_s$, just like above. Here making $N$ a power of 2 is even more important for the inverse FFT, and as above you keep $f_s$ equal to what the hardware requires. So you find $m_i'$ for each $f_i$ such that $m_i'$ is an integer, and find new frequencies $f_i'$ using $m_i'$ and equation $\ref{mint}$.

Before we start configuring the FPGA and RF Converter, first a little bit on radio theory might be in order.

Imagine you have 2 waveforms with different frequencies $\omega_1$ and $\omega_2$: $\cos\omega_1t$ and $\cos\omega_2t$ (but with the same amplitude). Multiplying these waveforms together is called "mixing" (as understood in the telecommunications industry), and the math is straightforward, using trigonometry identities: $$\begin{align} \cos([\omega_1+\omega_2]t)&=\cos\omega_1t\cos\omega_2t-\sin\omega_1t\sin\omega_2t \nonumber\\ \cos([\omega_1-\omega_2]t)&=\cos\omega_1t\cos\omega_2t+\sin\omega_1t\sin\omega_2t \nonumber\\ \sin([\omega_1+\omega_2]t)&=\sin\omega_1t\cos\omega_2t+\cos\omega_1t\sin\omega_2t\nonumber\\ \sin([\omega_1-\omega_2]t)&=\sin\omega_1t\cos\omega_2t-\cos\omega_1t\sin\omega_2t\nonumber \end{align}\nonumber$$ Adding and subtracting $\cos([\omega_1+\omega_2]t)$ and $\cos([\omega_1-\omega_2]t)$ gives: $$\begin{align} \cos\omega_1t\cos\omega_2t &=\frac{1}{2}\big[\cos([\omega_1-\omega_2]t)+ \cos([\omega_1+\omega_2]t)\big]\label{mixing}\\ \sin\omega_1t\sin\omega_2t&=\frac{1}{2}\big[\cos([\omega_1-\omega_2]t)- \cos([\omega_1+\omega_2]t)\big]\label{mixing2}\\ \end{align}$$ and adding and subtracting $\sin([\omega_1+\omega_2]t)$ and $\sin([\omega_1-\omega_2]t)$ gives: $$\begin{align} \sin\omega_1t\cos\omega_2t &=\frac{1}{2}\big[\sin([\omega_1-\omega_2]t)+ \sin([\omega_1+\omega_2]t)\big]\label{mixing3}\\ \cos\omega_1t\sin\omega_2t&=\frac{1}{2}\big[\sin([\omega_1+\omega_2]t)- \sin([\omega_1-\omega_2]t)\big]\label{mixing4}\\ \end{align}$$

Equation $\ref{mixing}$ (or any of these equations) tells us that when you mix 2 waveforms, the result is 2 waveforms, one which has the difference in frequency and one that has the sum. The actual multiplying of two analog waveforms together is not a trivial thing, and for analog waveforms you might have to build a special circuit using op-amps with matched diodes or transistors, but it's quite a common thing to do this!

Now let's say you have a wave $\cos\omega t$ (frequency $\omega$) that you want to send, and you want to send it on a "carrier wave" that has a larger frequency $\omega_0$ ($\omega_0\gt\omega$). Say both have the same amplitude. If the carrier wave is described by $\cos\omega_0t$ and you mix them together, you will get a transmitted waveform $$\cos\omega_0t\cos\omega t=\half\cos(\omega_0-\omega)t+\half\cos(\omega_0+\omega)t\nonumber$$ Note that if you mix the $\cos\omega t$ wave with $\sin\omega_0t$ instead of $\cos\omega_0 t$, you get $$\sin\omega_0t\cos\omega t=\half\sin(\omega_0-\omega)t+\half\sin(\omega_0+\omega)t\nonumber$$ Here you see clearly that the two transmitted waveforms consist of one with frequency $\omega_0-\omega$ and one with frequency $\omega_0+\omega$. The terminology is that you are sending information ($\cos\omega t$) in the "sidebands" of the carrier wave that is centered at $\omega_0$. What this also means is that the bandwidth needed to transmit information that consists of the frequency $\omega$ is $(\omega_0+\omega)-(\omega_0-\omega)=2\omega$. This is not an efficient use of bandwidth!

Let's say the transmitter mixes the incoming $\cos\omega t$ with a carrier wave $\cos\omega_0 t$. The transmitted wave will then be the product of the signal with the carrier: $$TX(t)=\cos\omega t\cos\omega_0 t\nonumber$$ The receiver then mixes the incoming signal with a tone from an oscillator that has the same frequency and phase as the carrier wave: $\cos\omega_0 t$. This gives you $$\begin{align} RX(t) &= TX(t)\cos\omega_0 t\nonumber \\ &= (\cos\omega t\cos\omega_0 t)\cos\omega_0 t\nonumber \\ &= \cos\omega t\cos^2\omega_0 t\nonumber \\ &= \half\cos\omega t(1+\cos 2\omega_0 t)\nonumber \\ &= \half\cos\omega t + \half\cos\omega t\cos2\omega_0 t\nonumber \\ &= \half\cos\omega t + \frac{1}{4}\big[ \cos(2\omega_0-\omega)t+\cos(2\omega_0+\omega)t\big] \end{align}\nonumber$$ You can see in the last step that what you are left with is the original signal $\cos\omega t$ (with half the amplitude) plus a signal centered at $2\omega_0$ with sidebands at $\pm \omega$. That last part is easily filtered by a low pass filter, leaving only the signal, at half the amplitude (because you've thrown half of the signal away with the filtering).

Now imagine that you want to send some information to someone over radio frequencies, encoded in some kind of time varying voltage that's not a simple $\cos\omega t$. For instance, say the information is of the form $f(t)$, which tells you the voltage as a function of time. $f(t)$ might be something simple like the waveform resulting from a microphone output, so the frequency range will be limited to within a few 10s of kHz. You can mix $f(t)$ with a carrier wave at a higher frequency, say $\cos\omega_0t$ where $\omega_0$ could be in the MHz radio range, and send all of the Fourier components of $f(t)$ mixed with $\cos\omega_0t$. The resultant waveform will have the Fourier components of $f(t)$ each mixed with $\cos\omega_0t$, so you will transmit the sum and the difference of each component with the carrier wave. At the receiver, you then generate the same carrier wave and mix it with the antenna output, and you will again get 2 waves: the difference, which will be $f(t)$, and the sum, which will be $f(t)$ mixed with twice the carrier wave. That higher frequency waveform is easy to filter out, leaving $f(t)$, which is what you want to send. Such is the power of the principle of superposition!

Voila, you've invented what is called homodyne radio transmission! (Note: heterodyne radio first mixes the incoming waveform down to an intermediate frequency (IF), mainly so that you can filter noise at a lower frequency than the high frequency carrier wave. It does this by mixing the incoming signal with a wave that has a frequency slightly different from the carrier wave, filtering the resultant IF waveform, and then mixing with another waveform that has the IF frequency to get the symbols.)

Back to the simple case of a carrier wave ($\omega_0$) and some information ($\omega$). The above works if you mix the wave with the information you want to transmit ($\cos\omega t$) with a carrier wave at the transmitter and the resulting wave with a carrier wave at the receiver, and both carrier waves have the same phase. In other words if you mix $\cos\omega t$ with $\cos\omega_0t$ at the transmitter, then you mix the resulting wave with $\cos\omega_0 t$ at the receiver. And same for if you use $\sin\omega_0t$, you have to use the same wave at the receiver. If you don't, and mix the $\cos\omega t$ with $\cos\omega_0t$ at the transmitter: $$TX(t)=\cos\omega t\cos\omega_0 t\nonumber$$ and $\sin\omega_0t$ at the receiver, you would get: $$\begin{align} RX(t)&=TX(t)\sin\omega_0 t\nonumber \\ &=\big[\cos\omega_0 t\cos\omega t\big]\sin\omega_0 t \nonumber \\ &= \cos\omega t\big[\cos\omega_0 t\sin\omega_0 t\big]\nonumber \\ &= \half\cos\omega t\sin 2\omega_0 t\label{cossin} \end{align}$$

This result eliminates the signal and leaves you with only the high frequency component, which when you filter out leaves you with nothing. In other words, the relative phase between the transmitter and receiver carrier wave matters greatly.

So what we really want to do is to construct things so that the receiver is "phase blind" and doesn't have to know what the phase of the transmitter carrier (whether it's $\sin$ or $\cos$). We would also like to decrease the required bandwidth of $2\omega$ for sending a signal at a carrier frequency of $\omega_0$. So to summarize, we now have 2 problems to solve here:

1. Mixing the $\cos\omega t$ wave with a carrier wave requires twice the bandwidth, since the result sent will be 1 wave with the difference in frequencies and one with the sum
2. The receiver needs to know the phase of the transmitter carrier wave to recover the signal

SSB transmission

Trigonometry is amazing, and the beginnings of the solution to the above 2 problems was due to an engineer working at General Electric in the 1940s and 1950s named Donald Norgaard, who pioneered the phasing method for single sideband communication. He designed hardware that split an analog signal into 2 paths, one phase shifted by $90^\circ$, and mixed them with 2 carriers that had a relative phase of $90^\circ$. So we start with a signal that has 2 phases: an "in-phase" component (I) and a "quadrature" component (Q), where "in-phase" is relative to the original signal, here $\cos\omega t$. The information wave would have the form: $$\begin{align} I(t)&=\cos\omega t\nonumber \\ Q(t)&=\sin\omega t\nonumber \\ \end{align}\nonumber$$ Then you mix the $I(t)$ component with the $\cos\omega_0 t$ carrier and the $Q(t)$ component with the $\sin\omega_0 t$ carrier, you would get 2 waves: $$I(t)\cos\omega_0 t = \cos\omega t\cos\omega_0 t = \half[\cos(\omega_0-\omega)t+\cos(\omega_0+\omega)t]\nonumber$$ and $$Q(t)\sin\omega_0 t = \sin\omega t\sin\omega_0 t = \half[\cos(\omega_0-\omega)t-\cos(\omega_0+\omega)t]\nonumber$$ and subtract them together to make the transmitted wave $TX(t)$: $$\begin{align} TX(t) &=I(t)\cos\omega_0 t-Q(t)\sin\omega_0 t\nonumber\\ &=\cos\omega t\cos\omega_0t-\sin\omega t\sin\omega_0t\nonumber\\ &=\cos(\omega_0+\omega)t\label{iq} \end{align}$$ By subtracting, we are choosing the upper sideband $\omega_0 + \omega$. If we were to add we would be choosing the lower sideband $\omega_0 - \omega$. This is just a convention, both are equally valid, but the point is that we now are sending only 1 sideband, decreasing the bandwidth needed for transmission. This shows clearly that we've already solved problem 1 above: we are now sending a signal that has a single frequency $\omega_0+\omega$, which means we need a smaller bandwidth than if we send one with the sum and one with the difference in these two frequencies.

At the receiver, you take this signal and split it into two equal parts by dividing the power, so each part will have an amplitude: $$\frac{1}{\sqrt 2}\cos(\omega_0+\omega)t\nonumber$$ Then you mix one of them with $\cos\omega_0 t$ ($I$) and the other with $\sin\omega_0 t$ ($Q$) . To see this explicitly: $$\begin{align} I &= \big[\frac{1}{\sqrt 2}\cos(\omega_0+\omega)t\big]\cos\omega_0 t\nonumber \\ &= \frac{1}{\sqrt 2}\big[\cos\omega t\cos\omega_0t-\sin\omega t\sin\omega_0t\big]\cos\omega_0 t\nonumber\\ &= \frac{1}{\sqrt 2}\big[\cos\omega t\cos^2\omega_0 t - \sin\omega t\sin\omega_0 t\cos\omega_0 t\big]\nonumber\\ &= \frac{1}{\sqrt 2}\big[\half\cos\omega t(1+\cos 2\omega_0t)-\half\sin\omega t\sin 2\omega_0 t\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[\cos\omega t+(\cos 2\omega_0 t\cos\omega t-\sin 2\omega_0 t\sin\omega t)\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[\cos\omega t+\cos(2\omega_0+\omega)t\big]\nonumber\\ \end{align}\nonumber$$ Voila, we recover the original transmitter $I(t)$ and filter out the higher frequency part near $2\omega_0$. The other half of the signal is mixed with $\sin\omega_0 t$ at the receiver to get: $$\begin{align} Q &= \big[\frac{1}{\sqrt 2}\cos(\omega_0+\omega)t\big]\sin\omega_0 t\nonumber \\ &= \frac{1}{\sqrt 2}\big[\cos\omega t\cos\omega_0t-\sin\omega t\sin\omega_0t\big]\sin\omega_0 t\nonumber\\ &= \frac{1}{\sqrt 2}\big[\cos\omega t\sin\omega_0 t\cos\omega_0 t - \sin\omega t\sin^2\omega_0 t\big]\nonumber\\ &= \frac{1}{\sqrt 2}\big[\half\cos\omega t\sin 2\omega_0 t-\half\sin\omega t(1-\cos 2\omega_0t)\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[-\sin\omega t+(\sin 2\omega_0 t\cos\omega t+\cos 2\omega_0 t\sin\omega t)\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[-\sin\omega t+\sin(2\omega_0+\omega)t\big]\nonumber\\ \end{align}\nonumber$$ So we recover what we sent.

This is exactly how the ZYNQ RF Converter works, except that it does it in the digital domain.

So far we assumed the receiver's oscillator has exactly the same phase as the transmitter's carrier. But problem 2 was precisely that the receiver should not have to know that phase. So let's let the receiver oscillator be off by some unknown phase $\phi$, mixing the incoming signal with $\cos(\omega_0 t+\phi)$ on the $I$ branch and $\sin(\omega_0 t+\phi)$ on the $Q$ branch. At the receiver, the steps are identical to above, just carrying $\phi$ along: $$\begin{align} I &= \Big[\frac{1}{\sqrt 2}\cos(\omega_0+\omega)t\Big]\cos(\omega_0 t+\phi)\nonumber\\ &= \frac{1}{2\sqrt 2}\big[\cos([\omega_0+\omega+\omega_0]t+\phi)+ \cos([\omega_0+\omega-\omega_0]t-\phi)\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[\cos(\omega t-\phi)+\cos((2\omega_0+\omega)t+\phi)\big]\nonumber \\ Q &= \Big[\frac{1}{\sqrt 2}\cos(\omega_0+\omega)t\Big]\sin(\omega_0 t+\phi)\nonumber\\ &= \frac{1}{2\sqrt 2}\big[\sin([\omega_0+\omega+\omega_0]t+\phi)- \sin([\omega_0+\omega-\omega_0]t-\phi)\big]\nonumber\\ &= \frac{1}{2\sqrt 2}\big[-\sin(\omega t-\phi)+\sin((2\omega_0+\omega)t+\phi)\big]\nonumber \end{align}\nonumber$$ After the low pass filter removes the terms near $2\omega_0$, the two branches are $$\begin{align} I &= \frac{1}{2\sqrt 2}\cos(\omega t-\phi)\nonumber \\ Q &= -\frac{1}{2\sqrt 2}\sin(\omega t-\phi)\nonumber \end{align}\nonumber$$

Now treat the two branches as the real and imaginary parts of a single complex number, which is what "$I+iQ$" really means. Recombining (the minus just absorbs the sign flip on the $Q$ branch we already noted above): $$I-iQ = \frac{1}{2\sqrt 2}\big[\cos(\omega t-\phi)+i\sin(\omega t-\phi)\big] = \frac{1}{2\sqrt 2}\,e^{i(\omega t-\phi)} = \frac{1}{2\sqrt 2}\,e^{i\omega t}\,e^{-i\phi}\nonumber$$

The transmitter sent the complex baseband $e^{i\omega t}$, and we recovered it multiplied by a constant phasor $e^{-i\phi}$. The unknown receiver phase $\phi$ does not destroy the signal, it simply rotates the $(I,Q)$ vector by a fixed angle. Contrast this with the single-mixer case of equation $\ref{cossin}$, where the wrong phase wiped the signal out completely.

And if you care about the signal power, exactly the situation when you're hunting for an axion conversion signal in ADMX, the phase drops out entirely: $$I^2+Q^2 = \frac{1}{8}\big[\cos^2(\omega t-\phi)+\sin^2(\omega t-\phi)\big]=\frac{1}{8}\nonumber$$ independent of $\phi$. The receiver is genuinely phase blind: by keeping both quadratures it recovers the full complex amplitude up to a known rotation, and the power $I^2+Q^2$ comes out intact regardless of the relative phase between the two oscillators. That solves problem 2.

You have a sine wave that has a peak-to-peak range of $V_{pp}$, and you want to sample it with an $N$ ADC at some sampling rate. The ADC can return $2^N$ different values, and each bit of the ADC covers a voltage range of $V_{pp}/2^N$. This is called the $LSB$. $$LSB \equiv \frac{V_{pp}}{2^N}\label{LSB}$$ If $V_{pp}=2$ volts (say the ADC is bipolar between $\pm 1$ volt) and $N_{bits}=8$, then $LSB=2/256=7.8$ mV. If the output of the ADC returns a value, call this $V_{adc}$ and the signal that gets sampled is called $V$, then your quantization error is defined as: $$\delta = \frac{V_{adc}-V}{LSB}\label{qe}$$

Because standard rounding quantizes to the nearest integer of the ADC, this "error" is really an uncertainty, and is bounded within $\pm 0.5 LSB$, which means that any value returned by the ADC is only accurate up to half an LSB.

In the following, we plot a $107.3 kHz$ sine wave quantized with an 8-bit ADC, showing the pure sine wave, and the white sampled points after quantization.

The noise is low, but there are frequencies outside of the main sine wave that are showing up in the Fourier plot, and that means that those frequencies are somewhat enhanced. This is a consequence of the quantization, because you are sampling at a fixed window in time, and the sine wave has a fixed period, so there will be repeatable places where the quantization error is large. And that will inject "noise" at a fixed rate, boosting the SFDR. And we might want to get rid of those "spurs" in the frequency spectrum.

So the way to deal with those spurs is to actually add noise, and do it in a way that will smear the power of those spurs into adjacent bins, bringing down the SFDR. But it will add to the noise floor.

Note that we want to add enough noise to the analog signal before quantization so that the quantization error pushes the quantized values past digitization boundaries. So if we added noise that has an RMS of less than 0.5 LSB, it will usually not due a whole log to change the digitized values from what they would be without dithering, so let's add noise that has an RMS of 0.8 LSB. The effect is shown in the next plot, where we see the difference between the digital values before and after dithering.

So what did dithering accomplish? It improved SFDR by 1.5 dB, and it raised the noise floor from -97.6 dBFS to -79.4 dBFS, which looks like we made the nose 18.2 dB worse. But more importantly, the spur that we see is not really a spur, it's a noise fluctuation, and it's 11.1 dB above the noise, as opposed to 30.8 dB, an improvement of over 20 dB. And, if you do the experiment again, you will find the maximum in a different place.

When you have an analog signal you want to digitize, and the amplitude is small compared to the LSB, standard quantization doesn't just create spurs, but instead turns the continuous sine wave into a harsh square wave or step function. Just imagine that the amplitude goes below 1 LSB, then it becomes a flat spectrum, even worse! With amplitude dithering, the noise constantly bounces the weak signal across the quantization threshold, and with averaging you can reconstruct the signal wherease without dithering you will get the signal and a lot of spurs at similar amplitudes.

In applications like radar, you absolutely do not want spurs since you can't tell those apart from real signals. So dithering is important there.

If the signal is so weak that it's at the same level as the spurs, then you can't tell the signal from the spur, but dithering will still help if the signal stays at a fixed frequency and the spurs average out to zero. With enough averaging the signal will remain.

The RFSoC board is basically implementing a digital version of homodyne radio transmitting and receiving: for transmission (using the DAC), you can supply a waveform $f(t)$, specify a carrier wave frequency (it will generate the carrier wave), and it will mix the two together and send it along. For receiving (using the ADC) you supply the carrier wave frequency and it will mix and filter and present you with the result.

The RF Data Converter on the ZU48DR (ZYNQ) chip groups 2 ADCs into what they call a tile, and the same for DACs (2 DACs per tile). This is done to save resources to some extent, as each of the 2 converters in a tile share the same clock and PLL resource, but the additional benefit of grouping into a tile is that because each converter in the tile runs off the same clock and PLL, they are inherently phase locked, and this can be important when generating signals or converting analog to digital. Also, the chip allows you to use 2 DACs in a tile to output a complex I and Q waveform, hence phase coherence will be important. For our purposes, we will only use the DAC to send out real waveforms.

Tiles that contain ADCs are numbers 224-227 and tiles that contain the DACs are numbered 228-231. These numbers may seem odd but they are simply the numbers of the I/O banks the converters connect to on the FPGA, and they are sequential to keep the clock routing simple (and minimize jitter). The ADCs and DACs are referred to by Vivado (2024.2) as "ADC 0" and "ADC 1" on a tile, and same for the DACs ("DAC 0" and "DAC 1"). If you look at the 4x2 board, you will see the ADCs and DACs labeled at the SMA connectors with names ADC_A, ADC_B, ADC_C, ADC_D, DAC_A, DAC_B. The mapping from SMA label to tile to Vivado label (not an easy thing to get as it involves quite a bit of cross-checking!) is the following:

The tiles you can use will be 228 and 230 for the DACs (with only 1 DAC available on each tile), and 224 and 226 for the ADCs (with both ADCs available on each tile). We will use DAC 0 on tile 228, which means DAC_B on the 4x2 output, and ADC 1 on tile 226, which will mean inputs are on ADC_A on the board.

Imagine that you want to create an analog output that consists of a carrier wave with some symbols mixed in. With a ~10 GSps DAC, by the Nyquist theorem you are limited to carrier waves at 5GHz or less. You could create the I and Q components in a Python array that lives in the DDR, and DMA the data to the FPGA and RF Converter and have it added together and digitized by the DAC and output. But if the DAC is digitizing at 10 GSps, then you'd have to clock your data in at that rate, and there's no FPGA that can handle data rates that fast. In fact, try running with a fabric clock above maybe 400 MHz on the ZU48DR and you will find it difficult to meet timing constraints, which means the design will be prone to setup/hold violations and the kind of hard-to-debug behavior (from things like race conditions) that comes with them. And the same goes for the ADC: digitizing at 5 GSps means I and Q data appear at that rate, and this is impossible to handle inside the FPGA.

The solution is to "up convert" from the digital data to the sampling rate. Up conversion is done by the "Digital Up Converter" (DUC) circuit, described next.

Digital Up Converter (DUC)

The DUC's job is actually twofold:

• Take complex data (that's what we will be using in this tutorial) that come in at some manageable rate that the FPGA can handle and increase the data rate to the DAC sampling rate
• Take the new I and Q higher rate data and perform the quadrature mixing with a carrier wave at a frequency $\omega_0$ to produce a real signal with the 1-sided band.

So for example, say you have some data representing a sine wave with a 50MHz frequency, and you want to convert it to an analog waveform using a DAC running at 400MSps. We will generate the data at 200MSps, comfortably above the Nyquist rate for a 50MHz signal, and let the DUC interpolate by 2x to bring it up to the 400MSps DAC conversion rate. In the figure, the markers show the coarse 200MSps data sent toward the DAC, and the line shows the sine wave that these markers fall on. You can see that the period is 20ns=1/50MHz and we are sending 4 samples per period, which is what a 200MSps data rate gives you.

And the Fourier spectrum will just be a spike in the 1st Nyquist zone. Note that we are Fourier analyzing the incoming data stream at a sampling rate that is the incoming data stream rate (200MSps). In the plot below, only the positive frequencies are shown. We see frequencies in the 1st 2 Nyquist zones: the 50MHz signal and the image at 200MHz-50MHz=150MHz.

Now for the interpolation. Since you are running the DAC conversion at twice the rate as the incoming "coarse" data, you need to produce data at twice the rate that you can feed into the converter. So what the DUC first does in "interpolating" is to increase the data rate by using each coarse data point and adding a zero in every other clock cycle. The following plot shows this new data stream superimposed on the incoming sine wave with the coarse data.

The following plot shows the 50MHz sine wave superimposed on the coarse data with zeros added:

Since you've added zeros in between each coarse data point, we now have a data stream with twice the number of points per second, so this is effectively sampling at twice the rate (and that higher rate is the DAC sampling frequency we are going to use). Also, those extra zeros mean we now have a data stream with quite a bit more transitions, so we've effectively added a higher frequency component to the data.

If we Fourier analyze this new data stream, since the rate is twice the rate of the coarse data, we are "sampling" at a higher rate , and the 1st Nyquist zone is now twice what it was for the Fourier analysis of the coarse data. The new frequency that shows up is exactly the image of the 50MHz incoming wave in the 2nd Nyquist zone. So by adding zeros you aren't really adding new frequencies, you are just increasing the sampling rate so that the images show up as harmonics in the new choppy (zero added) data stream. Having twice the sampling means that the 1st image from the 50MHz data now falls inside the analysis band. This is seen in the Fourier plot below:

Zero stuffing allows you to up convert to the DAC sampling rate, but it adds those higher frequencies. So you have to filter those higher frequencies out. What the DUC does then is to take the I and Q streams that have been zero stuffed (we increased the rate by x2, so our "interpolation" is 2), and apply a filter that filters out everything in the higher Nyquist zones (keeping only the 1st). This filtering is on each clock cycle of the new double rate data. For 2x interpolation the filter is a so-called "half-band" FIR, where FIR stands for "Finite Impulse Response" which describes a filter that uses a finite amount of the incoming data. To illustrate, start with an incoming data stream $y[n]$ where $n$ changes at some rate, and you want to up convert by x2. So conceptually, now you have an array $z[m]$ where $m$ changes at twice the rate of $n$, and every other value for $m$ is the zero you've added. You form the FIR which is a finite array $h[k]$ of size $L$. Now you apply the FIR to all values of $z[m]$ to make $z_{out}[m]$: $$z_{out}[m] = \sum_{i=-T}^{T} z[m-i]h[i]\label{FIR}$$ Each element of the filter is called a "tap", which runs from $-T$ to $T$. The $k=0$ element is called the "center tap", and you set that equal to 1, and every other tap that is an even number away from the center tap is set to 0. This way, when the filter lands on a non-zero data element, it just gets copied over as the filter output. When it lands on a zero element, it then uses the non zero data on either side of that zero to form the filter output. So a typical half-band FIR with 5 non-zero coefficient pairs (10), 5 zero pairs (10) plus the center tap, for a total of 21 taps, and might look like the following:

   h[-10] = +0.000000   (zero -- half-band)
   h[ -9] = +0.003783
   h[ -8] = -0.000000   (zero -- half-band)
   h[ -7] = -0.019580
   h[ -6] = +0.000000   (zero -- half-band)
   h[ -5] = +0.061492
   h[ -4] = -0.000000   (zero -- half-band)
   h[ -3] = -0.165014
   h[ -2] = +0.000000   (zero -- half-band)
   h[ -1] = +0.619393
   h[ +0] = +1.000000   <-- center tap
   h[ +1] = +0.619393
   h[ +2] = +0.000000   (zero -- half-band)
   h[ +3] = -0.165014
   h[ +4] = -0.000000   (zero -- half-band)
   h[ +5] = +0.061492
   h[ +6] = +0.000000   (zero -- half-band)
   h[ +7] = -0.019580
   h[ +8] = -0.000000   (zero -- half-band)
   h[ +9] = +0.003783
   h[+10] = +0.000000   (zero -- half-band)

There are also 3x and 5x interpolation stages we will meet shortly that don't have this pass-through property, and there, every output sample is a filtered combination. Also note that this means that there will be some delay (latency) inside the DUC due to the fact that it has to wait to accumulate the data points that come after each sample to use in the filter.

One more subtlety: stuffing zeros between the samples cuts the amplitude of the original signal in half, because the signal energy is now shared with the image. The interpolation filter therefore includes a compensating gain of 2 (a gain of $N_i$ in general for an interpolation factor $N_i$). That's why, after applying the filter, the data points in the figure below land back on the original 50MHz sine wave at full amplitude. The new data points fit pretty well on the original sine wave, as they should. This should not be a surprise: by applying what is effectively a low pass filter, if it filters out the higher frequencies then it also smooths out the data.

So the output of the DUC is a smoothed out stream of data that is at the higher rate: $f_{DAC} = N_i\cdot f_{in}$ where $f_{in}$ is the sample rate of the data driven into the DUC, $f_{DAC}$ is our DAC conversion rate, and $N_i$ is our interpolation factor. One caution here: $f_{in}$ is the input sample rate, which is not necessarily the same as the FPGA fabric clock frequency. The AXI4-Stream interface into the RF Data Converter normally carries more than one sample per fabric clock cycle (this is the "samples per AXI4-Stream cycle" setting in the Vivado IP configuration), so the fabric clock is $f_{DAC}/(N_i \times W)$ where $W$ is the number of samples per cycle. This is exactly what makes the high converter rates usable: with, say, 8x interpolation and 8 samples per cycle, a 6.4 GSps DAC only needs a 100 MHz fabric clock.

Note that the way this is implemented in the ZU48DR is pretty clever. Imagine that you want to use a large interpolation, like 40x. That would mean a very large FIR that will have quite a bit more than the 5 data samples on either side of the non-zero one, and the more samples you have, the more calculations you have to do, and that means more time and more power. So instead, they construct filters that only need to do 2x, 3x, 4x, and 5x. Then to get to 40x, they can apply these filters in series, each time increasing the interpolation by that factor. So 40x would be from applying 2x, 4x, and 5x in succession. The ZU48DR can apply up to 3 FIR filters in the last part of the DUC stage, with the first FIR (FIR1) able to do 2x, 3x, or 4x interpolation, FIR2 can do 2x, 3x, or 5x, and FIR3 can do 2x.

Each DUC and each DDC in the ZU48DR contains a complex mixer driven by its own "Numerically Controlled Oscillator", or NCO, which allows you to choose a programmable digital frequency $f_0$ with 48-bit resolution, with no analog synthesizer required. There is also a "coarse" mixer option that mixes at exactly $f_0=f_s/2$ or $f_0=\pm f_s/4$. At those special frequencies the local oscillator samples are just $0$ and $\pm 1$, so the mixer "multiplication" reduces to sign flips and I/Q swaps, costing essentially no power and introducing zero spurs. To see this explicitly, note that quadrature functions are sequences of numbers, so $I(t)\to I[nT_s]$ and $Q(t)\to Q[nT_s]$ where the time $t\to nT_s$ and $T_s = 1/f_s$. If we start with any signal $s(t)$, then to form $I$ we multiply the signal by: $$\begin{align} I[t] &= s(t)\cos(\omega_0 t)\nonumber\\ &= s(nT_s)\cos(2\pi f_0 n/f_s)\nonumber\\ &= s(nT_s)\cos(2\pi \frac{f_s}{2} n/f_s)\nonumber\\ &= s(nT_s)\cos(n\pi)\nonumber\\ &= (-1)^ns(nT_s)\nonumber \end{align}\nonumber$$ So all the mixer has to do is to flip the sign of every other $s(t)$. It's also easy to show that $Q[nT_n]=0$ since $Q$ is formed by multiplying by $\sin(\omega_0 t)$.

If you choose $f_0=f_s/4$ we get $I[nT_n]=s(t)\cos(n\pi/2)$ which means multiplying $s(t)$ by successive $+1, 0, -1, 0$ and repeat. And the same for $Q[nT_n]$. So there really is no need for any multiplication when using the coarse mixer for any of these 3 mixing frequencies.

Add ZYNQ

Click the thick "+" in the "Diagram" window, type "ZYNQ" in the Search window, and choose "Zynq UltraScale+ MPSoC" (the ARM processing system, the PS). Do not choose the one with "RF Data Converter" in the title yet, that is a separate IP we will add in a moment. Double click the "MPSoC" entry to drop it in, then click the blue "Run Block Automation" link, leave everything selected, and hit OK. Rename the block to "ZYNQ" in the "Block Properties" window.

While you have the ZYNQ block open, we need to make sure a PL clock is available to clock the memory-mapped and control sides of our logic. Double click the ZYNQ block, go to the "Clock Configuration" tab, expand "Output Clocks" → "Low Power Clock Domains" → "PL Fabric Clocks", and make sure PL0 is enabled (100 MHz is fine for now). This "PL0" clock will enable an output port called pl_clk0 on the ZYNQ block, and this port will clock the DMA's AXI4 and AXI4-Lite ports. The fast sample-rate clocks for the converters come from a completely different place (the on-board clock chips), which we will get to.

You should see "Run Block Automation" at the top of the "Diagram" window. Click that now. This adds important board preset capability, which configures the DDR4 controller, PS clocking, and wires up the clocks and resets. When you click it, it opens a new window called "Run Block Automation". Make sure "Apply Board Preset" is checked, and everything on the left is checked ("All Automation" etc) and hit OK. You should see this in the Diagram window:

Configuration summary

• Add "Zynq UnltraScale+ MPSoC" block
• Rename to "ZYNQ"
• Click on "Run Block Automation"

Add Constraints

Add a constraints file exactly as in the GPIO and DMA tutorials so the bitstream is compressed and the over-temperature protection is on. In the "Sources" window click the "+" symbol, select "Add or create constraints", hit Next, click "Create File", call it "Loopback", and click OK, then Finish. In the sources window, you should see the Constraints change to "Constraints(1)", which means the new file is there. Expand and you should see "Loopback.xdc". Double click to open and an editable window should appear to the right, where you add:

## compress the bitstream to make it smaller
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
# enable the over-temperature shutdown features
set_property BITSTREAM.CONFIG.OVERTEMPSHUTDOWN ENABLE [current_design]
#
# set LE0D to show the RX_FIFO reset condition
set_property -dict { PACKAGE_PIN AR11 IOSTANDARD LVCMOS18 } [get_ports { LED0 }]
# set LED3 for the heartbeat
set_property -dict { PACKAGE_PIN AU10 IOSTANDARD LVCMOS18 } [get_ports { LED3 }]

The first 2 constraints compress the bit file and enable over-temperature shutdown features, and the next 2 set up so that we use the leds on the top left of the board to show a few interesting things (talked about below).

Save the file (control-s).

We want to drive the DAC output with a cyclic DMA, but we want the ADC to be a one time write into DDR using the RX_FIFO reset to trigger. Cycle vs regular is best done with 2 different DMA engines, so we will need to add both here:

Add two "AXI Direct Memory Access" blocks exactly as in the DMA tutorial. One will be for the DDR4 to DAC path, and one will be for the ADC to DDR4 path, and they will have to be configured differently. To do this, go back to the Diagram window, click the plus sign to add a block, search for "AXI DMA", select "AXI Direct Memory Access", and double click. A new block appears, the title underneath whold be "AXI Direct Memory Access". Then add another (copy/paste should work but if not, do it by hand). Name one "AXI_DMA_SG" and the other "AXI_DMA_NOSG" just so we can differentiate in Python.

AXI_DMA_SG takes data from DDR4 and sends it towards the TX_FIFO, so we want the MM2S channel enabled, and disable the S2MM channel. The AXI_DMA_NOSG is the opposite - we are streaming into DDR4 so we want the S2MM channel enabled and MM2S disabled. Double click on the AXI_DMA_SG, and you will see both "Enable Read Channel" and "Enable Write Channel" enabled. In the 2024.2 version of Vivado, read and write is with respect to DMA, so MM2S is a read and S2MM is a write. Disabling the "Enable Write Channel" on AXI_DMA_SG disables S2MM, which is what you want, uncheck and click OK. Then double click on AXI_DMA_NOSG and disable the read channel and click OK.

Next we have to configure these 2 DMA enginesTo figure the "AXI_DMA_SG", double click to configure it:

• Make sure "Enable Scatter Gather Engine" is checked. This will be how we continuously send a waveform to the DAC
• Set the "Width of Buffer Length Register" to 26. This register holds the transfer byte count, and its width caps the largest single transfer at 2^width−1 bytes. The default of 14 allows only 2¹⁴−1 = 16383 bytes (~16 KB); 26 lets us transfer up to ~64 MB in one shot. (Because the DMA engine does not buffer (it moves data without storing it) this same byte count applies on both the memory side and the stream side of a transfer.)
• Towards the bottom you will see "Enable Read Channel" and "Enable Write Channel". The naming "read" and "write" is anchored on the memory operation: "read" means the DMA reads from DDR, so that's the MM2S path, and "write" means it writes into DDR, so that's the S2MM path. The AXI_DMA_SG engine will use the scatter-gather capability to cycle data from the DDR4 through to the DAC, so we want to enable the read channel (MM2S, should be the default) and disable the write channel (S2MM).
• Set the "Memory Map Data Width" to 32 (which should be the default), to match the converter's I/Q stream width we will create below, as each I/Q sample needs 32 bits (16 bits for I and 16 for Q), so a 32-bit bus delivers 1 sample per beat. We could actually go faster, but for this project, we are going to send, then receive, and compare, and there's no hurry. (But you could conceivably increase this from 32 to 256, which means that burst mode would send 8x the amount of data. The RF Converter is on the Stream side of the DMA engine, and that has to be 32 bits for the FIFO, if you set the memory mapped side to 256 then the AXI_DMA engine will take care of unpacking/packing for you.)

Click OK at the bottom to accept those settings.

For the AXI_DMA_NOSG engine, double click, unselect "Enable Scatter Gather Engine" and set the "Width of Buffer Length Register" to 26, and disable "Enable Read Channel" since here we will be writing from the ADC to the DDR4, which is considered a "Write Channel". Click OK to save.

Then click on the "Run Connection Automation" in blue at the top of the Diagram page. The default should be that all is selected on the left, but if not, select "All Automation". You should see that both the AXI_DMA_SG and AXI_DMA_NOSG are listed, and under each is "S_AXI_LITE". This is to hook up the AXI-Lite path, which is for controls. Hit OK. If after that you still see a "Run Connection Automation" at the top, do it again, connection automation is sometimes iterative, and this time it will probably want to hook up the AXI-Lite ports on the DME engines to the ZYNC (PS) chip. Click OK. Then click "Regenerate Layout", the button with the clockwise circle at the top of the Diagram page. The Diagram window should show this:

Configuration summary

• Add 2 "AXI Direct Memory Access" blocks
• Rename to "AXI_DMA_SG" and "AXI_DMA_NOSG"
• "AXI_DMA_SG"
  • Enable scatter gather
  • Change "Width of Buffer Length Register" to 26
  • Enable Read Channel and disable Write Channel
• "AXI_DMA_NOSG"
  • Disable scatter gather
  • Change "Width of Buffer Length Register" to 26
  • Enable Write Channel and disable Read Channel
• Run Connection Automation twice, enable everything

Probably a good time to save the project.

A "heartbeat" is just some way for you to be sure that the FPGA is alive and reasonably well. One classic way to do this is to just flash one of the LEDs on the board at a reasonable rate, like 1 Hz, so if it's flashing (the heart is beating) you know it's alive.

We will be setting the FPGA system clock to 250MHz below. To use this clock for a heartbeat, we will have to slow it down to about 1 Hz. The easiest way to think of how to do this this is do make a counter that is so large that the MSB toggles at around 1 Hz. If the system clock is 250MHz, then that clock changes state every 4ns, which means we need a counter that can count up to around $250\times 10^6$, which means the counter has to be arond 28 bits. So first let's make a verilog file that will do that:

module heartbeat (
    input  aclk,
    input  aresetn,   // active low
    
    output  heartbeat
);

reg [27:0] counter;
assign heartbeat = counter[27];

always @ (posedge aclk) 
    if (aresetn) counter <= counter + 1;
    else counter <= 28'b0;

endmodule

Create this file somewhere on the PC you are using to build the project and call it "heartbeat.v". In Vivado, click the "+" sign to add sources, select "Add or create design sources", click "Add Files" in the next window, navigate to this new file and select it, and then make sure "Copy sources into project" is checked. Then hit Finish. You should see a new file appear under "Design Sources" in the Sources tab.

Now go to the Diagram window, right click, and select "Add Module". You should see "heartbeat.v" in the list of sources, select it and hit OK. This creates a new block called "heartbeat_0". Rename it "HEARTBEAT", and connect "pl_clk0" on ZYNQ to "aclk" on HEARTBEAT, and connect "aresetn" on HEARTBEAT to "peripheral_aresetn" on the Processor System Reset block. Then right click on the Diagram window again and select "Create Port", give it the name "LED3", set Direction to Output and Type to Data and hit OK, and connect the "heartbeat" output of HEARTBEAT to this new port.

Configuration summary

• Add heartbeat.v to project sources
• Right click on diagram window, Add Module, select heartbeat.v
• Rename to "HEARTBEAT"
• Connect aclk to pl_clk0 on ZYNQ
• Connect aresetn to peripheral_aresetn on Processor System Reset block rst_ZYNQ_99M (or equivalent)
• Right click on diagram, Create Port
  • Port name: LED3
  • Direction: Output
  • Type: Data
• Connect heartbeat port on HEARTBEAT to LED3

We will need 3 GPIO blocks here:

The DMA delivers data from DDR in bursts, but the DAC drains its input at a continuous line rate, so to keep from starving the DAC at burst boundaries we put in an elastic buffer. The same goes for the other direction, taking data out of the ADC and writing it into DDR, so we need a buffer there too, and we will use FIFOs. So click on the "+" to add blocks, and search for "AXI FIFO", and click on "AXI4-Stream Data FIFO". Careful, not "AXI-Stream FIFO", that's a different beast, "AXI4-Stream Data FIFO" is the lightweight streaming FIFO, same one we used in the DMA tutorial. Double click to add. Then copy and past to add another. Rename one "TX_FIFO" and the other "RX_FIFO".

The names follow the usual radio convention: TX (transmit) is the outbound path, where data flows from DDR out to the DAC, and RX (receive) is the inbound path, where digitized data flows from the ADC back into DDR. Lining that up with the DMA (whose "read" and "write" are both with respect to DDR): TX_FIFO sits in the MM2S path (DMA reads DDR, streams out), so its output feeds the DAC input of the RF converter. RX_FIFO sits in the S2MM path (DMA writes DDR), so its input is driven by the ADC output of the converter.

For each FIFO, double click and make the following changes in the "Re-customize IP" window:

• Set a reasonable depth (for example 1024).
• Change the data width ("TDATA width") from the default to 4 bytes (32 bits). This is under "Signal properties" at the bottom; you first have to change the access from AUTO to MANUAL.
• Set "Independent clocks" set to YES, as we will can use these FIFOs to cross the clock domains from the FPGA fabric clock (which is set at 100MHz) to whatever the DAC/ADC needs, determined by the sampling rate and whatever decimation or interpolation we will be using (see below).
• Make sure "Enable packet mode" is set to No. A "packet" could be the entire amount of data you want to transfer, and that will be larger than the 1024 FIFO depth, so you definitely don't want the FIFO to try to hold it until it gets a signal to release it!

Each FIFO has an S_AXIS port and an M_AXIS port. TX_FIFO is the transmitting data into the DAC, and that will be cyclic so we will use AXI_DMA_SG. So the TX_FIFO S_AXIS port connects to the M_AXIS_MM2S port on the AXI_DMA_SG block, and the TX_FIFO M_AXIS port will connect to the right port on the RF Converter (see below).

For the RX_FIFO, it is sending data from the ADC to the DDR, no cyclic, so it goes through AXI_DMA_NOSG, so you connect the M_AXIS port on RX_FIFO to the S_AXIS_S2MM port on AXI_DMA_NOSG, and you connect the S_AXIS port on RX_FIFO to the right port on the RF Converter to accept data from the ADC (see below).

So the next thing to do is to make these connections:

• Connect M_AXIS_MM2S on AXI_DMA_SG to S_AXIS on TX_FIFO (careful, that's M_AXIS_MM2S and not M_AXI_MM2S)
• Connect S_AXIS_S2MM on AXI_DMA_NOSG to M_AXIS on RX_FIFO. Note that as explained below, we are going to have to actually sever this connection and add something in series between the RX_FIFO output and the AXI_DMA input. But connect it for now.

Now let's connect the RX_FIFO_RESET line to the RX_FIFO reset. We could just connect the line by hand, but there's another consideration: a FIFO is a very synchronous object (that is, it does things by clock) whereas GPIO is asynchronous (it does things whenever you tell it via GPIO, which is not necessarily synchronous with any clocks). So we should take care. Now, when we issue the reset, that will clear the FIFO and prevent any new data from entering. So we don't have to worry about anything being synchronous. But when we release the reset, the FIFO will start up, and we don't want that to happen near the clock edge or we could get race conditions, which different flip-flops might change differently, which generates unknown effects. So we are going to have to synchronize our GPIO with the RX_FIFO write clock (which we will get to later), because that's the clock that defines "t=0" in the FIFO.

The way to do this is to instantiate a new block called "Processor System Reset". (One of these is already in the project, wired in when you hit "Run Connection Automation" for the ZYNQ block but this one you will instantiate will be it's own and not used for anything else.) Click on "+" in the Diagram window and type "Processor System Reset", double click, and rename it to "RX_FIFO_RESET_SYNC".

Then connect the following signals from "RX_FIFO_RESET_SYNC" to "RX_FIFO":

• "slowest_sync_clk" to "s_axis_aclk" on RX_FIFO, since "s_axis_aclk" will be the RX_FIFO write clock (slave port accepts data from the RF Converter). Note that both of these ports are inputs, and we haven't gotten to what drives those (see below).
• "aux_reset_in" to the RX_FIFO_RESET output (the AXI GPIO) output (it's the same port that connects to the LED0 output). This will be the reset trigger.
• "ext_reset_in" to the "pl_resetn0" line on the ZYNQ (PS) block. This is the global reset that you want to go to all elements.
• "peripheral_aresetn" connects to the RX_FIFO "s_axis_aresetn" line. This resets the RX_FIFO.

What this block is built to do is precisely what we need: at the rising edge of "slowest_sync_clk", "peripheral_aresetn" follows "aux_reset_in". So it's basically just a flip-flop. Here's the timing diagram, below. Remember, the thing we want to be sure of is that when the GPIO transitions from 0 to 1, that's the release of the FIFO reset, and we want to be sure that that line is synchronous with the clock.

There's two more lines to connect on the "RX_FIFO_RESET_SYNC" block: the line "dcm_locked" has to be set to 1 and "mb_debug_sys_rst" set to 0. The former is the signal that says the input clock is good, but there's no way to generate that since that clock will come from the RF Converter, so we set it to 1. The latter is a reset that we won't be using, so set it to zero. To do this, you instantiate 2 "Constant" blocks, rename one of them "ZERO" and the other "ONE", set the width to 1 bit and "ZERO" to 0 and "ONE" to 1 and tie them in.

Go ahead and hit the "Run Connection Automation" link at the top. If you've followed without any mistakes or omissions, you should see the options are to connect RX_FIFO/m_axis_aclk, RX_FIFO_RESET_SYNC/slowest_sync_clk, RX_FIFO_TLAST/GPIO, and TX_FIFO/s_axis_aclk. All of these are going to be connected by hand, so you don't want Vivado to route them because then you just have to undo it. If you have any other options, then probably it's ok to enable them and let Vivado route it, so it OK. But if not, then hit Cancel.

Hit the "Regenerate Layout" button. You should see a block diagram that looks like this:

Configuration summary

• Add 2 "AXI4-Stream Data FIFO"
• Rename to "RX_FIFO" and "TX_FIFO"
• For both, IP Configuration:
  • FIFO depth: 1024
  • Independent Clocks: Yes
  • TDATA width (bytes): 4
• Connect M_AXIS on RX_FIFO to S_AXIS_S2MM on AXI_DMA_NOSG
• Connect S_AXIS on TX_FIFO to M_AXIS_MM2S on AXI_DMA_SG
• Add "Processor System Reset"
  • Rename to "RX_FIFO_RESET_SYNC"
  • Connect slowest_sync_clk to s_axis_aclk on RX_FIFO
  • Connect aux_reset_in to RX_FIFO_RESET output
  • Connect ext_reset_in to pl_resetn0 on ZYNC block
  • Connect peripheral_aresetn to s_axis_aresetn on RX_FIFO
  • Add 2 constant blocks. Rename to "ONE" and "ZERO"
  • Change "ZERO" Const Val to 0
  • Connect output of "ZERO" to mb_debug_sys_rst on RX_FIFO_RESET_SYNC
  • Connect output of "ONE" to dcm_locked on RX_FIFO_RESET_SYNC
• Don't click Run Connection Automation!

Next we want to enable the ports on the PS (the ZYNQ block) so we can do DMA from the DDR4 to the AXI_DMA engine, and we should increase the FPGA fabric clock so that we are sure we are reading the RX_FIFO faster than we are writing it, just to leave us some comfort!.

Now we want to connect the AXI bus (not the AXIS stream side!) lines between the ZYNQ ports we just made and the AXI_DMA ports. And there are actually 3 ports on the AXI_DMA block that we need to connect: M_AXI_SG (gets scatter-gather info), M_AXI_MM2S (grabs data from DDR and sends it to the TX_FIFO), and M_AXI_S2MM (takes data from RX_FIFO and sends it to DDR). So there are 3 AXI master ports on AXI_DMA and 2 AXI slave ports on the PS to access DDR. Since AXI is a complicated thing, we always want to route lines from master to slave on AXI through an AXI Smartconnect block. This block will handle all of the conversions necessary, and keeps things in order.

So back to the Diagram, click "+" and search for Smartconnect, and add 2 "AXI SmartConnect" blocks. Open up one of them and in the configuration, change "Number of Slave Interfaces" to 1 and hit OK, and rename that block "SMART_NOSG". Rename the other AXI Smartconnect to "SMART_SG", and make sure it has 2 slave ports.

The MM2S direction is where the AXI_DMA_SG engine pulls data from DDR4 and sends it into the RFDC DAC input, cycling. So that one will have the MM2S direction and the scatter-gather ports on AXI_DMA_SG. AXI_DMA_NOSG will connect to SMART_NOMSG. So make the connections that are outlined in the following table. And be careful to connect M_AXI_* and not M_AXIS_*, we are connecting on the AXI DDR side of the DMA engine!

Then by hand connect "aresetn" on both of these SmartConnects to the "peripheral_aresetn" output on the rst_ZYNQ_99M Processor System Reset block.

Don't "Run Connection Automation", as we are not finished connecting things by hand!

Hit "Regenerate Layout". Your block diagram should now look something like this:

This would be a good time to save the project.

Configuration summary

• Add 2 "AXI SmartConnect" blocks. Rename to "SMART_SG" and "SMART_NOSG".
• Double click "SMART_NOSG" and change "Number of Slave Interfaces" to 1
• SMART_SG connections:
  • S00_AXI to M_AXI_MM2S on AXI_DMA_SG
  • S01_AXI to M_AXI_SG on AXI_DMA_SG
  • M00_AXI to S_AXI_HP0_FPD on ZYNQ
• SMART_NOSG connections:
  • S00_AXI to M_AXI_S2MM on AXI_DMA_NOSG
  • M00_AXI to S_AXI_HP1_FPD on ZYNQ
• Connect "aresetn" on both of these SmartConnects to the "peripheral_aresetn" output on the rst_ZYNQ_99M Processor System Reset block.
• Don't click Run Connection Automation

Before working on the RF Converter, there's one more issue. When we want to read data from the RX_FIFO into DDR4, we ask the DMA engine to to initiate the transfer, all through Python. So the code knows the size of the buffer, but this size is really a maximum as far as the DMA goes: don't send more than this into the buffer. But the DMA engine is not counting data that gets sent, it's only sending data to the DDR4 using the AXI protocols, which involve the following control signals:

• TVALID This is used by the bus master to tell the slave that "I have valid data ready for you on TDATA"
• TREADY This is the slave telling the master "I am ready to accept data now".
• TLAST This is the master telling the slave "this specific data sample is the last one of the current packet"

Data only moves across the bus when both TVALID and TREADY are high at the same time. This is how they "handshake".

The problem is, who sends TLAST? For the transfer from DDR4 to the AXI DMA engine to the TX_FIFO to the DAC, the AXI DMA engine itself generates TLAST. But when data goes the other way, from the FIFO into the DMA engine, there's no way to know how much data to then send into the DDR4. So no one generates TLAST and the system hangs.

But TLAST is known by the system and actually is sent to the DMA engine on the AXI_LITE bus as a slow control. So we will have to make our own module that sits in series between the RX_FIFO M_AXIS port and the AXI_DMA S_AXIS_S2MM port that gets this information, and when the transfer starts, counts, and asserts TLAST when it's done.

Click the "+" button in the block design window to add another IP, search for ZYNQ and add "Zynq Ultrascale+ RF Data Converter". Change the name to "RFDC" in the "Block Properties" window.

Note that the "Run Block Automation" and "Run Connection Automation" links are still there, which is good. The "Run Block Automation" is for the RFDC that you just instantiated, it sets it with what works with the RealDigital 4x2 board. So this should be run first. But it's safer to first configure the RF Converter for the ADCs and DACs you want to use, before "Run Block Automation" just to be sure it doesn't configure something you have to change later.

Double click on the RFDC module you just instantiated. It will bring up a window that will look like this:

You will see that the "RF-ADC" tab is selected, and you see the 4 tiles 224, 225, 226, and 227 are tabbed below. Let's use "ADC_A" on our 4x2 board, and that input is connected to "ADC 1" of tile 226, so unselect "Enable ADC" under "ADC Tile 224", then click on "ADC Tile 226" and click "Enable ADC" under "ADC 1".

Next click on "RF-DAC". Let's enable the "DAC 0" in tile 228, and that one comes out the "DAC_B" SMA connector on the 4x2. So click on "tile 226" and enable "DAC 0", and make sure "DAC 1" on that tile and no other DACs are enabled (they shouldn't be). Then hit "OK". This will change the RFDC block, which should now look like this:

You will see these inputs and outputs:

• Inputs:
  • s_axi The AXI-LITE control path
  • s00_axis The AXI stream (AXIS) data input to the DAC path
  • adc2_clk The 491.52 MHz reference clock that drives the PLL for the tile that contains the ADC. Vivado will create the right port and connect it for you.
  • dac0_clk Same clock as adc2_clk.
  • vin2_23 The analog signal from the ADC_A SMA connector that goes into the ADC
  • sysref_in A differential signal used to keep things aligned.
  • s_axi_aclk The clock for the AXI-Lite path
  • s_axi_aresetn The reset for the AXI-Lite path
  • m2_axis_aresetn The reset signal for the ADC output path that sends data into the RX_FIFO
  • m2_axis_aclk The clock that goes with the aresetn above
  • s0_axis_aclk The clock that drives the data from TX_FIFO into the DAC path
  • s0_axis_aresetn The reset signal for the DAC input path digital data
• Outputs:
  • m22_axis 16-bit I output from the ADC
  • m23_axis 16-bit Q output from the ADC
  • vout00 The analog outout to the DAC_B SMA connector from the DAC
  • clk_adc2 The clock that is used to create the m2_axis_aclk input
  • clk_dac0 The clock that is used to create the s0_axis_aclk input
  • irq Unused

On these inputs and outputs, the "s_" and "m_" prefixes refer to "slave" and "master", and this is from the point of view of the RFDC block relative to the FPGA fabric. So "m_" means that the RFDC drives it, meaning an ADC conversion of the analog input that delivers digital data from the ADC out of the RFDC into the FPGA. "s_" means that the FPGA sends data into the RFDC so that it can convert it at the DAC to an analog output.

We are done configuring the RFDC. Click ok.

Configuration summary

• Add "Zynq UltraScale+ RF Data Converter" block to Diagram window
• Rename to "RFDC"
• Double click
• Select "Basic" tab
  • Select "RF_ADC" tab and deselect all ADCs for all 4 tiles except "ADC 1" on "ADC Tile 226"
  • Select "RF_DAC" tab and deselect all ADCs for all 4 tiles except "DAC 0" on "DAC Tile 228"
• Change "AXI4-Lite Clock (MHz)" to 250
• Select "System Clocking" tab at the top
  • Enable "PLL" for "ADC 226" and "DAC 228"
  • Change Sampling rate to 6.88128 for DAC 228 and 4.9152 for ADC 226
  • Change "Reference Clock (MHz)"" to 491.52 for both ADC 226 and DAC 228
• Select "Basic" tab
• Select "RF_ADC"
  • Go to "ADC Tile 226" ADC 1
  • Disable "Dither"
  • Digital Data Ouput: I/Q
  • Samples per AXI4-Stream Cycle: 1
  • Mixer Type: Fine
  • Mixer Mode: Real->I/Q
  • Decimation Mode to 40x
  • Nyquist Zone: 1
  • Calibration Mode: Mode 2
• Select "RF_DAC"
• Go to "DAC Tile 228" DAC 0
  • Enable "Inverse Sinc Filter"
  • Analog Output Data: Real
  • Interpolation Mode: 40x
  • Samples per AXI4-Stream Cycle: 2
  • Mixer Type: Fine
  • Mixer Mode: I/Q->Real
  • Nyquist Zone: 1
  • Decoder Mode: SNR Optimized
• Select "System Clocking" tab at the top
  • Change "Clock Out (MHz)"" for ADC 226 to 76.8
  • Change "Clock Out (MHz)"" for DAC 228 to 107.52
• Connect "M_AXIS" on TX_FIFO to "s00_axis" on RFDC

Double click on RX_FIFO_CLK, and click on the "Clocking Options" tab. At the bottom, in the row that has "Primary", in the "Input Frequency" column, change the setting to "Manual" and type in 76.8 MHz. Then click on the "Output Clocks" tab and change the "Output Freq" column in the "clk_out1" row (which should be enabled) to 122.88. At the bottom of that window, deselect "reset" and "locked" and hit OK. The RX_FIFO_CLK decal should change to just having a "clk_in1" and "clk_out1" port.

Then connect "clk_adc2" from RFDC to "clk_in1" of RX_FIFO_CLK. The output clock "clk_out1" of RX_FIFO_CLK has to go to several places:

• Connect to the "s_axis_aclk" port on RX_FIFO, since this is the input side of the FIFO, data flowing to it from the RFDC through the packer. When you connect it, since you've already connected "slowest_sync_clk" on RX_FIFO_RESET_SYNC to the RX_FIFO "s_axis_aclk" port, that will show up in the connection.
• Connect it to the "aclk" input on the packer if it's not already connected.
• Connect it to the "m2_axis_aclk" line on RFDC, since this drives the ADC output into the packer. Also connect the "m2_axis_aresetn" line on the RFDC to the "aresetn" line on the packer and connect both of those to "peripheral_aresetn" on RX_FIFO_RESET_SYNC so that that reset is synchronous with the clock that runs things between the RFDC and the RX_FIFO.

The following table summarizes the clocks and connections.

One more thing to configure. The RFDC receives data on the "s00_axis" port from the TX_FIFO. And we've connected the "s0_axis_aclk" on RFDC to the "m_axis_aclk" on TX_FIFO, but we still have to connect the reset line, "s0_axis_aresetn" on RFDC to something. In principle we can connect it to any resetn line and just have the reset be asynchronous, but that's a risk, and it has to do with when the reset is deasserted (going from reset to not reset). If that reset is not synchronous with the 172.032 "s0_axis_aclk" line on the RFDC, then there could be race conditions that can have unknown effects in the RFDC internals. So best to synchronize that reset with that clock. To do that, we add another reset block like before: click "+" and search for "reset" and add "Processor System Reset" and make the following connections:

• "slowest_sync_clk" connects to "clk_out1" on TX_FIFO_CLK
• "ext_reset_in" to "pl_resetn0" on ZYNQ
• "peripheral_aresetn" to "s0_axis_aresetn" on RFDC

The reset can be unconnected.

Configuration summary

• Add axis_iq_packer.v to project sources, right click Add Module in Diagram, select axis_iq_packer.v and rename to "AXIS_IQ_PACKER"
• Connect "m_22_axis" and "m_23_axis" from RFDC to "s_axis_i" and "s_axis_q" on AXIS_IQ_PACKER respectively
• Connect "m_axis" on AXIS_IQ_PACKER to "S_AXIS" on RX_FIFO
• Connect "aclk" and "aresetn" on AXIS_IQ_PACKER to "s_axis_aclk" and "s_axis_aresen" on RX_FIFO
• Add 2 "Clocking Wizard" blocks, rename "TX_FIFO_CLK" and "RX_FIFO_CLK"
• Configure TX_FIFO_CLK
  • Clocking Options tab: Primary "clk_in1" set "Input Frequency" to 107.52
  • Output Clocks tab: change "clk_out1" under "Requested" to 172.032
  • Deselect "reset" and "locked" at the bottom
• Configure RX_FIFO_CLK
  • Clocking Options tab: Primary "clk_in1" set "Input Frequency" to 76.8
  • Output Clocks tab: change "clk_out1" under "Requested" to 122.88
  • Deselect "reset" and "locked" at the bottom
• Connect "clk_dac0" on RFDC to "clk_in1" on TX_FIFO_CLK
• Connect "clk_out1" on TX_FIFO_CLK to "s0_axis_aclk" on RFDC and "m_axis_aclk" on TX_FIFO
• Connect "clk_adc2" on RFDC to "clk_in1" on RX_FIFO_CLK
• Connect "clk_out1" on RX_FIFO_CLK to "s_axis_aclk" on RX_FIFO, "aclk" on AXIS_IQ_PACKER (might already be connected), "m2_axis_aclk" on RFDC
• Connect "m2_axis_aresetn" on RFDC to "aresetn" on AXIS_IQ_PACKER and connect those to "peripheral_aresetn" on RX_FIFO_RESET_SYNC if not already connected
• Add "Processor System Reset" block to Diagram and connect
  • "slowest_sync_clk" to "clk_out1" on TX_FIFO_CLK
  • "ext_reset_in" to "pl_resetn0" on ZYNQ
  • "peripheral_aresetn" to "s0_axis_aresetn" on RFDC
• Click on Run Block Automation

Now we can click on "Run Block Automation". It will bring up a window, and if everything has been done right, the only thing it will need to do is to set up the reference clock inputs and the analog input and output for the RFDC. So you should see "RFDC" on the left, it should be enabled but if not, enable it. Block automation should take care of all required clocking connections depending on the configuration and options selected below, which tells you what this block automation does. There should be 2 options:

ADC 2 AXI-Streaming Clock Source
DAC 0 AXI-Streaming Clock Source

Next we delve into how to control all of this using Python in a Jupyter notebook. The first thing to make sure you do is to copy the .bit and .hwh file to the ARM chip where your notebook lives, and you do that by clicking the upload button (shown inside the red square below).

quiesce_pl: matched 2 of 4 named DMA(s) in /home/xilinx/jupyter_notebooks/ADMX/tutorials/Loopback_v1.bit
  AXI_DMA_NOSG @ 0xa0000000
    AXI_DMA_NOSG.MM2S        was_running=False halted=True reset_cleared=True  ok  [was already stopped]
    AXI_DMA_NOSG.S2MM        was_running=False halted=True reset_cleared=True  ok  [was already stopped]
  AXI_DMA_SG @ 0xa0010000
    AXI_DMA_SG.MM2S          was_running=False halted=True reset_cleared=True  ok  [was already stopped]
    AXI_DMA_SG.S2MM          was_running=False halted=True reset_cleared=True  ok  [was already stopped]
hwh_check: Loopback_v1.hwh -- no dangling interfaces or floating inputs
Firmware version:  0xa5000001
PS Clock:  249.9975  MHz
DAC: sampling at 6881.28 MHz and interoplation 40
ADC: sampling at 4915.2 MHz and decimation 40
all done

The command gen.ip_dict will type out the Python dictionary for all of the blocks defined in the project using Vivado.

  ▼ ip_dict:
  ▶ AXI_DMA_SG:
  ▶ AXI_DMA_NOSG:
  ▶ RX_FIFO_RESET:
  ▶ RX_FIFO_TLAST:
  ▶ VERSION:
  ▶ RFDC:
  ▶ ZYNQ:

Expand these and you will see all the attributes of each of these objects, including status and control registers.

DAC tile 0 (phys 228): Fs = 6881.2800 MSps, blocks enabled = [0]
    block 0: InterpolationFactor = 40, fabric = 172.0320 MSps, NCO = 99.99999999998721 MHz
ADC tile 2 (phys 228): Fs = 4915.2000 MSps, blocks enabled = [1]
    block 1: DecimationFactor = 40, fabric = 122.8800 MSps, NCO = 99.99999999999417 MHz

DAC: 6881.2800 MSps / 40x  -> 172.0320 MSps fabric
ADC: 4915.2000 MSps / 40x  -> 122.8800 MSps fabric
DAC sampling is consistent (6881.28)
DAC interpolation is consistent (40)
ADC sampling is consistent (4915.2)
ADC decimation is consistent (40)

and then calls them to put the system into a known state, ready to start the DAC path transfer. Code is below.

"""
set up the GPIOs for the RX_FIFO_RESET, RX_FIFO_TLAST over DMA, and vanilla DMA
"""
#
# setup for the RX_FIFO_RESET
#
rx_fifo_reset_ip = gen.ip_dict["RX_FIFO_RESET"]   # this setups up the pointer to the address in the dictionary
rx_fifo_reset = AxiGPIO(rx_fifo_reset_ip).channel1 # we are using channel 1 of the 2 channels
def RX_FIFO_RESET():
    rx_fifo_reset.write(0,0x1)
def RX_FIFO_RELEASE():
    rx_fifo_reset.write(1,0x1)
#
# now for TLAST
#
tlast_ip = gen.ip_dict["RX_FIFO_TLAST"]
tlast = AxiGPIO(tlast_ip).channel1
def SET_TLAST(n):
    tlast.write(n,0xFFFFFF)
#
# setup for DMA 
#
dma_tx = gen.AXI_DMA_SG
dma_rx = gen.AXI_DMA_NOSG

def DMA_SS2M_SOFT_RESET():
    dma_rx.register_map.S2MM_DMACR.Reset = 1
    t0 = time.time()
    print("DMA_SS2M_SOFT_RESET: S2MM status register: "+hex(dma_rx.register_map.S2MM_DMASR))
    while dma_rx.register_map.S2MM_DMACR.Reset:
        if time.time() - t0 > 0.1:
            raise RuntimeError("DMA soft reset did not complete!")
        time.sleep(0.001)
    print("S2MM control register: "+hex(dma_rx.register_map.S2MM_DMACR))
    print("S2MM status register: "+hex(dma_rx.register_map.S2MM_DMASR))

#
# set the RX_FIFO to be in the reset state
#
RX_FIFO_RESET()
#
# reset the S2MM DMA just to be sure we are in a known state
#
DMA_SS2M_SOFT_RESET()

Carrier Frequencies

Next, set the carrier frequencies, also known as the NCO (numerically controlled oscillator) values, in MHz. We will use 500MHz so that we can compare with the Xilinx documentation (see below).

"""
set the mixer frequencies for the ADC and DAC.  these should be the same for a loopback.  
these should be far enough away from the baluns limit of round 20 MHz
"""
gen.freq_adc = adc_carrier
gen.freq_dac = dac_carrier

Note that setting the frequencies like this is possible courtesy of the "@property" code above.

Contiguous Space

As a check, make sure you know how much contiguous free space you have:

!cat /proc/meminfo | grep -i cma

The exclamation in front of the command allows Python to submit it to the system. This should return something like:

CmaTotal:    524288 kB
CmaFree:     513320 kB

Note that our DAC sampling is given in MHz, so we have to multiply by $10^6$ to get Hz, which is what Python is expecting for frequencies. Also, note the lines where we find the number of cycles in the array m, turn it into an integer, and then calculate the frequency f that will be used to sample to make the array values. Then the code makes the i_a and q_a I and Q arrays. I've also commented out 2 lines below that that you can uncomment so that you just send a constant into the DAC. The output should be:

DAC sampling at 6.881280 GSps
DAC 'fabric' rate at 172.032 MHz
modulation tone at 5000000.00000000 Hz = 5.0 MHz
Number of cycles is set to 60952
Modified modulation tone at 4999968.75000000 Hz = 5.0 MHz,  differs by 31.250 Hz
Array duration is 0.0121905 sec
Took 0.657 seconds to fill both arrays

As you can see from the output, the 5MHz modulation tone is modified by 31.250Hz so that there's no phase discontinuity between cycles.

We use np.rint to round to the nearest integer (np.floor rounds downwards and would slightly distort symmetrical AC waveforms) and cast using astype(np.int16) instead of just np.int16 because the latter will make Numpy first copy the array into another array before moving into the buffer.

At this point, if you take the DAC output and put it into a spectrum analyzer, you should see a nice peak like in the following photo, where the width of the peak is measured to be a few Hz, which might be at the limit of the instrument.

Followed by the resulting output:

BlockStatus:  {'SamplingFreq': 4.9152, 'AnalogDataPathStatus': 1, 'DigitalDataPathStatus': 897, 'DataPathClocksStatus': 1, 'IsFIFOFlagsEnabled': 3, 'IsFIFOFlagsAsserted': 0}
Mixer settings: {'Freq': 499.99999999998835, 'PhaseOffset': 0.0, 'EventSource': 2, 'CoarseMixFreq': 0, 'MixerMode': 3, 'FineMixerScale': 0, 'MixerType': 2}
DSA: {'DisableRTS': 0, 'Attenuation': 0.0}
TLAST target set to 0x10000

-----------------
Starting run
CalibrationMode: 2   Dither: 0
S2MM DMASR armed: 0x0
BlockStatus after RX_FIFO_RELEASE:  {'SamplingFreq': 4.9152, 'AnalogDataPathStatus': 1, 'DigitalDataPathStatus': 897, 'DataPathClocksStatus': 1, 'IsFIFOFlagsEnabled': 3, 'IsFIFOFlagsAsserted': 0}
Will wait  1.0  seconds for the data
nonzero words: 131060 of 131072
ADC data arrived ok

-----------------
Starting run
CalibrationMode: 2   Dither: 1
S2MM DMASR armed: 0x1000
BlockStatus after RX_FIFO_RELEASE:  {'SamplingFreq': 4.9152, 'AnalogDataPathStatus': 1, 'DigitalDataPathStatus': 897, 'DataPathClocksStatus': 1, 'IsFIFOFlagsEnabled': 3, 'IsFIFOFlagsAsserted': 0}
Will wait  1.0  seconds for the data
nonzero words: 131065 of 131072
ADC data arrived ok

"""
    Analyze!

    this next command has to do with the fact that writing INTO DDR from the FPGA
    does not go through the ARM chip, but directly to DDR through the HP ports in
    the PS.  but the ARM chip might still hold state lines for some of those memory
    locations you allocated (and would hold zeros).  calling invalidate() for the
    receive buffer marks them dead so the next numpy read refetches and doesn't use
    ARM cache.

    Note: the spectrum x-axis is the baseband offset from the NCO frequency 
    (±61.44 MHz for our 122.88 MSps I/Q rate), not the absolute RF frequency. 
    To plot absolute RF, add the NCO frequency to the axis (and mirror the sign 
    if the input is in the second Nyquist zone).
"""
#
# check if we got anything by counting the number of non-zero elements.  if it's 0,
# of much less than the number of data points received, then bail.
nonzero_off =  np.count_nonzero(recv_ADC_buffer_ditheroff)
print("Dither off run:")
if nonzero_off == 0:
    print("Dither off:nonzero words:", nonzero_off, "of", len(recv_ADC_buffer_ditheroff))
    raise SystemExit("Something happened for dither off run, no data arrived.  Bail!")
else:
    print("Number of non-zero words ",nonzero_off," so data is probably ok")
print("Dither on run:")
nonzero_on =  np.count_nonzero(recv_ADC_buffer_ditheron)
if nonzero_on == 0:
    print("Dither on:nonzero words:", nonzero_on, "of", len(recv_ADC_buffer_ditheron))
    raise SystemExit("Something happened for dither on run, no data arrived.  Bail!")
else:
    print("Number of non-zero words ",nonzero_on," so data is probably ok")
#
# now dig the dither on and off data out of the arrays
#
print("\nDither off data:")
i_rx_off = recv_ADC_buffer_ditheroff[0::2].astype(np.float64)
q_rx_off = recv_ADC_buffer_ditheroff[1::2].astype(np.float64)
iq_data_off = i_rx_off - 1j * q_rx_off
iq_off_max = abs(iq_data_off).max()
iq_off_min = abs(iq_data_off).min()
print(f"  iq_off min {iq_off_min:.1f}  iq_off_max {iq_off_max:.1f} counts w/full scale 32767")
iq_off_scale = 0.5*(iq_off_max + iq_off_min)
#
# 16 bit data is 16 bits with the upper 14 bits being from the ADC, in 2's complement, so
# full scale is 2^16 or +- 2^15
#
iq_off_attenuation = iq_off_scale/32767.0
iq_off_attenuation_db = 20*math.log10(iq_off_attenuation)
print(f"  attenuation {iq_off_attenuation:.3f} = {iq_off_attenuation_db:.1f} dB")

print("\nDither on data:")
i_rx_on = recv_ADC_buffer_ditheron[0::2].astype(np.float64)
q_rx_on = recv_ADC_buffer_ditheron[1::2].astype(np.float64)
iq_data_on = i_rx_on - 1j * q_rx_on
iq_on_max = abs(iq_data_on).max()
iq_on_min = abs(iq_data_on).min()
print(f"  iq_on min {iq_on_min:.1f}  iq_on_max {iq_on_max:.1f} counts w/full scale 32767")
iq_on_scale = 0.5*(iq_on_max + iq_on_min)
iq_on_attenuation = iq_on_scale/32767.0
iq_on_attenuation_db = 20*math.log10(iq_on_attenuation)
print(f"  attenuation {iq_on_attenuation:.3f} = {iq_on_attenuation_db:.1f} dB")
#
# type out some useful information
#
f_fabric = rfdc_adc_sampling / rfdc_adc_decimation           # MHz
N_off = len(iq_data_off)
N_on = len(iq_data_on)
print("Array length:  off = ",N_off,"  on = ",N_on)
rbw_hz_off = f_fabric * 1e6 / N_off
rbw_hz_on = f_fabric * 1e6 / N_on
print("\nDither off:")
print(f"Captured {N_off} pairs at {f_fabric} MHz -> {N_off/f_fabric:.1f} us, "f"RBW = {rbw_hz_off:.1f} Hz")
#
# process both dither on and off data.  fft_process does the FFT and analyze_fft_noise digs out
# NSD and other useful quantities
#

do_window = True
fft_db_off = fft_process(iq_data_off,do_window)
freqs_off = np.fft.fftshift(np.fft.fftfreq(N_off, d=1.0 / f_fabric))   # MHz, offset from NCO
results_off = analyze_fft_noise(fft_db_off,iq_off_scale,1.0E6*rfdc_adc_sampling,rfdc_adc_decimation, 32767,
                               enbw=1.5 if do_window else 1.0)
results_off_noise_floor = results_off['noise_floor_per_bin_dbc']
print(adc_report(results_off, "Dither off"))
n = len(fft_db_off)
fs_out = 1.0E6*f_fabric
for k, db in results_off["spurs"][:10]:
    f_mhz = (k - n // 2) * fs_out / n / 1e6     # offset from NCO
    print(f"{f_mhz:+10.4f} MHz   {db:7.2f} dBc")
enob_off = results_off['enob_intrinsic']



print("\nDither on:")
print(f"Captured {N_on} pairs at {f_fabric} MHz -> {N_on/f_fabric:.1f} us, "f"RBW = {rbw_hz_on:.1f} Hz")

fft_db_on = fft_process(iq_data_on,do_window)
freqs_on = np.fft.fftshift(np.fft.fftfreq(N_on, d=1.0 / f_fabric))   # MHz, offset from NCO
results_on = analyze_fft_noise(fft_db_on,iq_on_scale,1.0E6*rfdc_adc_sampling,rfdc_adc_decimation, 32767,
                               enbw=1.5 if do_window else 1.0)
print(adc_report(results_on, "Dither on"))
results_on_noise_floor = results_on['noise_floor_per_bin_dbc']
n = len(fft_db_on)
fs_out = 1.0E6*f_fabric
for k, db in results_on["spurs"][:10]:
    f_mhz = (k - n // 2) * fs_out / n / 1e6     # offset from NCO
    print(f"{f_mhz:+10.4f} MHz   {db:7.2f} dBc")
enob_on = results_on['enob_intrinsic']

# 
# make the plots
#
noise_indices = np.where(results_off["noise_mask"])[0]
fig_off = go.Figure()

# 1. Full FFT Spectrum (Gray line)
fig_off.add_trace(
    go.Scatter(
        y=fft_db_off,
        mode="lines",
        name="FFT Spectrum",
        line=dict(color="gray", width=1),
        opacity=0.5
    )
)

# 2. Isolated Noise Bins (Blue dots)
fig_off.add_trace(
    go.Scatter(
        x=noise_indices,
        y=fft_db_off[results_off["noise_mask"]],
        mode="markers",
        name="Noise Bins Used",
        marker=dict(color="#1f77b4", size=3)
    )
)

# 3. Horizontal line for average noise floor
fig_off.add_hline(
    y=results_off["noise_floor_per_bin_dbc"],
    line_dash="dash",
    line_color="red",
    annotation_text=f"Avg Noise Floor ({results_off['noise_floor_per_bin_dbc']:.1f} dBc)",
    annotation_position="bottom right"
)

# 4. Layout styling
fig_off.update_layout(
    title="FFT Spectrum & Automated Noise Floor Masking",
    xaxis_title="FFT Bin Index",
    yaxis_title="Power (dBc)",
    template="plotly_white",
    hovermode="x unified",
    height=500
)

fig_off.show()
fig0 = go.Figure()
fig0.add_trace(go.Scatter(x=freqs_off, y=fft_db_off, mode="lines",name="Dither off"))
fig0.update_layout(title=f"Dither off f_mod {1.0e-6 * f:.1f}MHz, f_NCO {adc_carrier:.1f}MHz, ENOB {enob_off:.1f}")
fig0.update_xaxes(title_text="Offset from NCO (MHz)")
floor = np.floor((fft_db_off.min() - 5) / 10) * 10        # from the data, not -100
fig0.update_yaxes(title_text="Normalized Amplitude (dBc)",range=[floor, 5])
fig0.add_hline(y=results_off_noise_floor,line_dash="dash",line_color="firebrick",line_width=1.5,
    annotation_text=f"noise floor {results_off_noise_floor:.1f} dBc/bin",
    annotation_position="top left",annotation_yshift=10)
fig0.show()

fig1 = go.Figure()
fig1.add_trace(go.Scatter(x=freqs_on, y=fft_db_on, mode="lines",name="Dither on"))
fig1.update_layout(title=f"Dither on f_mod {1.0e-6 * f:.1f}MHz, f_NCO {adc_carrier:.1f}MHz, ENOB {enob_on:.1f}")
fig1.update_xaxes(title_text="Offset from NCO (MHz)")
floor = np.floor((fft_db_on.min() - 5) / 10) * 10        # from the data, not -100
fig1.update_yaxes(title_text="Normalized Amplitude (dBc)",range=[floor, 5])
fig1.add_hline(y=results_on_noise_floor,line_dash="dash",line_color="firebrick",line_width=1.5,
    annotation_text=f"noise floor {results_on_noise_floor:.1f} dBc/bin",
    annotation_position="top left",annotation_yshift=10)
fig1.show()

Here is the printed output, chopped into blocks since there is a lot of it since both dither off and dither on are being analyzed. The first part just gives you the non-zero words, and checks that it's close to the total number expected. If so then "data is probably ok". If the number of non-zero words is 0, then it will let you know and bail. After that it tells you the min and max, which should be pretty close, and using full scale of $\pm 32767$ it tells you the attenuation, which should be around -17.7 dB.

Dither off run:
Number of non-zero words  131060  so data is probably ok
Dither on run:
Number of non-zero words  131065  so data is probably ok

Dither off data:
  iq_off min 4357.5  iq_off_max 4358.0 counts w/full scale 32767
  attenuation 0.133 = -17.5 dB

Dither on data:
  iq_on min 4356.0  iq_on_max 4355.1 counts w/full scale 32767
  attenuation 0.133 = -17.5 dB
Array length:  off =  65536   on =  65536

Next, the data with dither off is analyzed. It reports the number of pairs (I and Q) captured, the frequency, and the resulting frequency bin width RBW. You can see here that $\Delta f=1875$Hz as expected. Then the RF-ADC noise analysis. The noise floor is calculated (see below) by averaging over all of the cells in the normalized Fourier spectrum, which was made with a Hann window. The average noise floor is $-107.98$ dBc relative to the peak bin, and $-109.74$ dBc relative to the carrier's true power. The 1.76 dB difference is the ENBW correction: the Hann window spreads the carrier's power across a main lobe about 1.5 bins wide, so the single peak bin under-represents the carrier by that factor. The noise needs no such correction, white noise is flat per bin with or without the window.