Written as a retrospective…

A bug that I had picked up was to remove what was described to me as ‘high frequency ringing’ in the output of a derivative (determining the rate of change between recorded values) calculation, this calculation is performed in the frequency domain.

To add a layer of complexity: The input signal is the output of another process. This other process can have a mask applied to it, reducing the amount of data is calculated. This means that there can be a block of data in the middle of our signal, and then rest of the signal has zero value on either side of the data. You could imagine that this block of data has ‘hard edges’. As there is data in the middle of signal, we don’t expect to see results that are longer \ outside of the input data, but that’s exactly what can we have.

Our resulting data looks horrible, with artefacts in it. Not good.

There are two issues at play here.

- The ringing going past the input data edges when it has a mask applied
- High frequencies being boosted within the signal

And the solutions?

- Find where the data starts and ends, and mirror both ends of the signal
- Apply a filtering on a vector that relates to frequencies, this vector is used in the derivative calculation

## Solutions in some more detail

### The beginning, the end, and mirrors

As we can have data in the middle of a signal, which contains hard edges, we want to taper our signal. But we don’t want to taper real parts of the signal, otherwise we have an untrue result. Finding the start and end points are easy; iterate through the signal and record the first and last non-zero value.

For the mirroring, we need to know how much of the data is going to be tapered. When we know what percentage of the signal will be tapered, we can determine how long to make our mirrors so that the leading taper stops at the start of our real signal, and the trailing taper starts at the end of our real signal.

The the results are then manipulated to reconstruct an output signal with the same data beginning and end points, and we then don’t have the high frequencies shooting up and down from where the real data is.

### Filtering out the frequencies

When a derivative is performed in the frequency domain, it dampens the amplitude of the lower frequencies and boosts the amplitude of the higher frequencies.

We don’t want that to happen, hence the filtering.

Firstly, we want to determine from what frequency on wards should be attenuated, then we work backwards. As an example, I don’t want to see frequencies greater than 80Hz. I first need to find the index in the vector that corresponds¹ to 80Hz, decide on some kind of tapering length, and start tapering from index of 80Hz – tapering length.

The taper is a trailing taper, we used cosine squared, to have a nice effect.

Bringing these two techniques together results in much cleaner results when visualised.

#### Bringing the solutions together

The solutions are neatly coded up into nice modular functions, with unit testing. It’s all integrated into the software, allowing for interactive filtering \ attenuating of a user specified frequency if the end user wants to have filtering turned on. It’s resulted in a pretty slick experience for the user, something I’m proud of being able to produce.

#### Summary

I admit that this all sounds high level stuff, but that’s because I don’t really understand the maths involved with tapering, how the derivative is being performed in the frequency domain, etc… I’ve had plenty of help from my clever Mathematician colleagues, plus I’ve read a lot of the code where there is pre-existing functionality.

I’ve not even touched upon what needs doing to the data before the derivative calculation is performed; there’s preparation needed on the data before Fourier Transforms are executed. Nor the testing and prototyping involved; producing tests based upon analytical signals with known derivatives, testing calculating index in the vector of a given frequency, etc…

This has been a really good learning curve in some very interesting topics, aided by this fantastic resource: https://jackschaedler.github.io/circles-sines-signals/

I’ve not really done this post justice by omitting before and after examples.

¹ Or, as close to, if I’m being lazy and not using some kind of interpolation – which in fairness interpolation has not been implemented.