Proffieboard V2.2 suddenly going mute

Hello there,
I am currently installing a neopixel saber for my brother with a Proffieboard V2.2. So far hardware installation and customizing ProffieOS 6.7 were really straight forward. However, today I started doing some stability tests on the saber chassis outside of the hilt and noticed that the sound suddenly goes mute while doing some fast swings. To reproduce the issue, I extend the blade and do several fast consecutive swing motions. Then the sound goes mute. The blade or at least the neopixel leds on the internal chassis connector continue to function and the saber can be controlled by both the pow and aux button. If I retract the blade using the pow button and then, after it fully retracted, push the pow button again, the saber seems to work within normal parameters. At first, I was suspecting a hardware issue, but then I noticed that the issue does not occur with one of the three sound fonts, I installed on the saber. All three sound fonts worked until now on another Proffiboard v2.2 based saber. Looking at the output in the serial monitor revealed nothing suspicious, but I have a feeling that some property of the sound font might be causing this issue. Maybe an expected but missing file, or some specific property of the sound fond wav or config files. I already tried to get more information by enabling the diagnostic and developer commands in the config and using the monitor command, but have not found a usable clue on the source of the issue. Do you have some useful hints on how to debug this?

Best regards,
DeFi

Several hours of debugging later, I have strong indications to believe that the issue is related to the low frequency cutoff filter. I had enabled this filter in my config via

#define FILTER_CUTOFF_FREQUENCY 100
#define FILTER_ORDER 8

to get rid of some crackling noises due to a clearly overdriven speaker during smooth swing sounds overlying (although I only use it at a volume of 250 for testing purposes at the moment). If the filter is active, I am also able to reproduce the issue with the factory config in default_proffiboard_config.h and the default sound fonts. It does, however not occur as fast as on my custom sound fonts. If I do not activate the filter, I have not been able to reproduce the issue. Maybe there is a bug in the filter implementation or the resulting CPU load of the filter has some unforeseen side effects. Unfortunately deactivating the filter seems to be no option, since the otherwise occurring crackling noises sound like they have quite a potential to damage the speaker.

Could you specify exactly what font causes it?
You mention default config and default sounds.
Is it possibly smoothswing pair01in TeensySF?
Could you try this default package instead and see what results?

Additional testing suggests that it does occur apparently with any font that includes saber swing sounds. I was able to reproduce it at least with with TeensySF, SmthJedi, SmthFuzz, RgueCmdr with the default_proffiboard_config.h only extended by the above mentioned two lines for filter activation. The issue occurs quite randomly in normal operation and is hard to reproduce. If you want to trigger it, I suggest you take a light saber chassis, hold it in the center between thumb and middle finger and wiggle it back and forth at high speed. This way, I can reproduce the issue within seconds. In my last tests with the default_proffiboard_config.h, I discovered a possibly related issue: The saber I am building has three blades(two for accents and stuff). The default_proffiboard_config.h has only a neopixel and a three CREE blade configured. When I try to trigger the issue as described above, sometimes my second accent LED fires up switches to full white and remains stuck there until I retract the blade. I am starting to suspect that the Proffiboard starts triggering all kind of strange stuff, when under heavy load. Maybe some buffer overflows in the code or multithreading madness.

Interesting and unusual. I think some other people use the filters without hitting such issues.

Here is a list of possibilities:

  1. the volume has been turned to zero somehow
  2. A bug in the filters is causing it to produce little or no output
  3. The amplifier is cutting out because of a short or something (and resetting when disabled.)
  4. Magic, cosmic radiation, gremlins

A few serial monitor commands that can help:

a) get_volume If it got muted somehow, this would show it.
b) dacbuffer (requires ENABLE_DEVELOPER_COMMANDS) this will show a small set of the samples that are being sent to the amplifier, run this after having triggered the bug, but before retracting. If it’s a filter issue, this is expected to show all zeroes.
c) beep If this works, but not other sounds, then the problem is in the wav-playing code, not in the actual filter.
d) amp on / amp off if this helps, then the problem is probably (3) above.

1 Like

Hello! Thank you for your fast and expedient answer.

a) get_volume outputs 250, which is my configured VOLUME value
b) the command dacbuffer does not seem to exist. So I scanned the ProffieOS code and found the command buffered. It outputs:

Unit 0 Buffered: 512
Unit 1 Buffered: 0
Unit 2 Buffered: 512
Unit 3 Buffered: 512
Unit 4 Buffered: 0
Unit 5 Buffered: 0
Unit 6 Buffered: 0

I also found the command dacbuf. It outputs:

STATUS: 30000 CR1: 31280 CR2: 2
Current position: 57
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0

c) beep (lol, nice close encounters reference) does not work, but works again once the blade has been retracted
d) amp on / amp off output nothing in the serial monitor and do not change the muted sound

Looks like my previous observation that it might be connected to the filter was not too far off.

Small addition: while the sound is muted, the serial monitor still shows output like:

Playing Obi/hum/hum01.wav
channels: 1 rate: 44100 bits: 16
Playing Obi/swingl/swingl08.wav

So, it certainly thinks it is playing audio files.

This would certainly point to the filters (or something in that vincinity) doing something weird.

I can add some code to inspect the internal filter values.
It’s easier for me to add it to the github master, do you know how to get and test code from there? (If so, would you mind just trying it and see if it has the same problem or not.)

Getting the code from github is no problem. I just had a closer look at filters.h and dac.h. Could be filter_.clear(); in the begin function of LS_DAC getting mysteriously called a second time. However, the filter related code in that section did not look suspicious on first sight.

filter_.clear() shouldn’t make everything silent though, unless you call it a LOT.
It just clears out the state that persists between samples in the filter.

My guess is that something ends up as a NaN[1] for some reason.
Adding a command that prints out all internal variables in the filter should tell us if that’s what’s going on. It’s possible that filter_.clear() would actually make it work again if that is the case.

It’s weird though, because the filter code is almost exclusively additions and multiplications, which usually don’t create NaNs. Normally only divisions by zero and square roots on negative numbers create NaNs.

In the meantime I found another command that might help us figure out what is going on: monitor samples This command will print out some statistics from the dynamic mixer twice per second, which might be interesting.

[1] NaN = Not A Number.

Output of monitor samples during the error state:

Samples: 1218096 AVG volume: 299258 last input sample: 1676 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1227028 AVG volume: 168512 last input sample: -56 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1235872 AVG volume: 295518 last input sample: -124 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1244716 AVG volume: 189298 last input sample: -99 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1253560 AVG volume: 285727 last input sample: -1914 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1262492 AVG volume: 303998 last input sample: 374 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1271292 AVG volume: 278593 last input sample: -1498 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1280180 AVG volume: 165697 last input sample: -1108 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1289068 AVG volume: 180446 last input sample: 1846 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1297956 AVG volume: 160729 last input sample: 1331 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1306800 AVG volume: 276465 last input sample: 982 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1315688 AVG volume: 175099 last input sample: -349 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1324532 AVG volume: 219265 last input sample: -2539 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1333420 AVG volume: 247488 last input sample: 1253 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1342308 AVG volume: 250162 last input sample: 1124 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1351196 AVG volume: 268325 last input sample: 230 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1360040 AVG volume: 152707 last input sample: 159 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1368840 AVG volume: 228801 last input sample: -450 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1377728 AVG volume: 144409 last input sample: -1148 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1386572 AVG volume: 289913 last input sample: -1502 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1395416 AVG volume: 339971 last input sample: 437 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1404348 AVG volume: 330928 last input sample: 2016 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1413192 AVG volume: 334864 last input sample: -282 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1422124 AVG volume: 216351 last input sample: 187 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1430968 AVG volume: 361744 last input sample: -785 last output sample:0 peak sum: 0 peak: 0 underflows: 0
Samples: 1439812 AVG volume: 211982 last input sample: 97 last output sample:0 peak sum: 0 peak: 0 underflows: 0

Usually, I would find negative input samples and a constant output sample suspicious, but the output looks the same before the issue occurs.

Well, I have a theory.
It seems like at very high internal volumes, the volume calculated by the internal mixer could overflow. This would cause the square root to produce NaNs, which would propagate into the filter and cause problems until clear is called.

I’m adding a filterdata command which will print out the data values inside the filter. It won’t exactly explain where the problem comes from, but if we see NaN values in there, I will try making some modifications to the dynamic mixer to fix it.

Actually, the fix might be as simple as changing int32_t to uint32_t on this line:

With an uint32_t, even if it did overflow, no NaNs would be produced.

1 Like

In all instances I’ve seen of this, I’m pretty sure it’s always occurred with peak volumes. I wish I could remember where, or know where to search a previous instance or two to confirm that.

Looks like my theory is incorrect.
The numbers shouldn’t be able to add up enough to cause an overflow.
It’s still possible that a NaN is causing these issues somehow, but if so, I think it must be coming from somewhere else…

Good news: I tried your data type fix and until now, I was not able to reproduce the issue. If it still persists, it must take significantly longer to trigger it. Of course this still is no proof of it being a complete fix, so It would be nice tracing the issue to its origin. I will do some debugging this afternoon, but your filterdata debug command could come in handy in the future anyways. Regardless of that, I will report, if the issue shows up again. Thank you for the efforts, you put into this.

As for the unexpected LED triggering described earlier. I think this was only due to the second blade not being configured as neopixel at the time of testing. So the corresponding data pin could have been floating and the neopixels data line got fed with some uncontrolled garbage.

I just inserted some debug code into int read(float* data, int elements) which gets called instead of the integer version when FILTER_CUTOFF_FREQUENCY is defined. Just before sqrtf is called, I tracked the current and previous value of vol_ and the sum v. The following shows two occurrences of the issue in the serial monitor

unit = 4 vol = 0.50, Playing Obi32/swng/swng11.wav
channels: 1 rate: 44100 bits: 16
vol_prev: 8383571 v: 54980 vol: -8371629
vol_prev: -8371629 v: 54830 vol: -8284312
vol_prev: -8284312 v: 54635 vol: -8197530
vol_prev: -8197530 v: 54364 vol: -8111357
vol_prev: -8111357 v: 53894 vol: -8025989
vol_prev: -8025989 v: 53230 vol: -7941616
Audio underflows: 19
unit = 5 vol = 0.50, Playing Obi32/swng/swng15.wav
vol_prev: 8380179 v: -49031 vol: -8380933
vol_prev: -8380933 v: -48649 vol: -8299737
vol_prev: -8299737 v: -48283 vol: -8219222
vol_prev: -8219222 v: -47140 vol: -8140160
vol_prev: -8140160 v: -46039 vol: -8062504
vol_prev: -8062504 v: -45371 vol: -7985817
vol_prev: -7985817 v: -44923 vol: -7channels: 1 rate: 44100 bits: 16
Audio underflows: 17
unit = 3 vol = 0.00, Playing Obi32/swingl/swingl06.wav

…

channels: 1 rate: 44100 bits: 16
vol_prev: 8372485 v: -51134 vol: -8386502
vol_prev: -8386502 v: -52304 vol: -8301643
vol_prev: -8301643 v: -52711 vol: -8216710
vol_prev: -8216710 v: -52494 vol: -8132325
vol_prev: -8132325 v: -52193 vol: -8048569
vol_prev: -8048569 v: -51964 vol: -79vol_prev: -6983041 v: -59898 vol: -6896100
vol_prev: -6896100 v: -60027 vol: -6809370
vol_prev: -6809370 v: -60197unit = 2 vol = 0.00, Playing Obi32/swingl/swingl08.wav
channels: 1 rate: 44100 bits: 16

As can be clearly seen in all cases, the subterm ((vol_ + abs(v)) * 255) in line 122 overflows 2^31 because of the multiplication with 255, consequently feeding values smaller than zero into the square-root, which then cause an inescapable negative number frenzy. So, declaring vol_ as a uint32_t could indeed be a valid fix for this issue. :slight_smile: However, I do not know if it would be desirable to prevent such high values before this happens.

So my theory is right, even though I managed to convince myself that it can’t happen. Guess I’ll have to read the code again and see if I can figure out where I went wrong.

One possible way to fix this would be to replace this:

vol_ = ((vol_ + abs(v)) * 255) >> 8;

with:

vol_ += abs(v);
vol_ -= (vol_ + 255) >> 8;

Should be exactly the same thing, but without the multiplication by 255, and only costs one extra operation.

Hmm, I do not fully understand what the line does in terms of the filter (did not have time to read the paper) and I might interpret the rounding effects of right shift division wrong, but I have the impression the 255 should not be there:

vol_ += abs(v);
vol_ -= vol_ >> 8;

I guess there is no easy way to detect if the filter is working correctly, aside from the overflow bug.

x >> 8 is the same thing as x / 256
(x + 255) / 256 or (x + 255) >> 8 means “divide by 256 and round up”.

Without rounding up, vol_ won’t go down to zero. Once it reaches 255, division by 256 will become zero, and vol_ will stay at 255.

The original function doesn’t work like this, If vol_ + abs(v) is 255, then multiplying by 255 and dividing by 256 will result in 254, not 255.

Now, the 255 might not be necessary, but it’s not a random bug, it’s meant to be there.

Ah, I did not take the changed position of the rounding down shift division in the term into account. Thank you for enlightening me. I just tested the modified code and it seems to work.