Mass Storage is acting up on me

NoSloppy · November 29, 2021, 4:47am

MacOS 10.14.6 Mojave. ProffieOS6beta, Proffieboard v3.7.

While I’ve seen weirdness in the past as far as files showing over USB, but not via card reader, I couldn’t put my finger on the process to recreate what was happening, and there was also strange filename corruption. However, no corruption today, just complete missing files.

For some reason, I see all files normally, then see none at all depending on how the SD is mounted. While it’s certainly possible that the issue is my computer OS, what’s odd is that this has ALSO happened to at least 2 other people using Mass Storage and they are using Windows.
So I thought i’d bring it up as it seems it might NOT be computer OS related.

Mass Storage enabled, copied over a font. BlakStar for example.
Confirm copy is complete by listening to a wav or 2 from the font ON THE SD card.

Also list files in Terminal to see they’re there. (Yes I’m aware of the invisible resource files, not an issue ever before and I believe unrelated here)

BCmini-2:~ bconner$ cd /Volumes/PB37B/BlakStar 
BCmini-2:BlakStar bconner$ ls -al
total 8384
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 11:14 .
drwxrwxrwx@ 1 bconner  admin   32768 Nov 28 23:23 ..
-rwxrwxrwx@ 1 bconner  admin    6148 Nov 23 18:04 .DS_Store
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._.DS_Store
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._bgndrag
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._bgnlock
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._blst
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._boot01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._clsh
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._drag01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._enddrag
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._endlock
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._font01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._hum01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._in01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._lock01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._menter.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._mexit.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._mselect.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._out01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._slsh01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._spin01.wav
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._stab
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._swingh
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._swingl
-rwxrwxrwx  1 bconner  admin    4096 Nov 28 23:14 ._swng
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 03:36 bgndrag
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 03:52 bgnlock
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 00:46 blst
-rwxrwxrwx@ 1 bconner  admin  138064 Oct  3  2019 boot01.wav
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 05:40 clsh
-rwxrwxrwx@ 1 bconner  admin  733232 Oct  3  2019 drag01.wav
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 05:40 enddrag
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 05:43 endlock
-rwxrwxrwx@ 1 bconner  admin  138064 Oct  3  2019 font01.wav
-rwxrwxrwx@ 1 bconner  admin  651584 Jan  9  2020 hum01.wav
-rwxrwxrwx@ 1 bconner  admin   65978 Jan  8  2020 in01.wav
-rwxrwxrwx@ 1 bconner  admin  702284 Oct  3  2019 lock01.wav
-rwxrwxrwx@ 1 bconner  admin   81632 Oct  3  2019 menter.wav
-rwxrwxrwx@ 1 bconner  admin   74384 Oct  3  2019 mexit.wav
-rwxrwxrwx@ 1 bconner  admin   54414 Oct  3  2019 mselect.wav
-rwxrwxrwx@ 1 bconner  admin  139632 Oct  3  2019 out01.wav
-rwxrwxrwx@ 1 bconner  admin   40024 Oct  3  2019 slsh01.wav
-rwxrwxrwx@ 1 bconner  admin   53036 Oct  3  2019 spin01.wav
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 08:29 stab
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 10:56 swingh
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 10:55 swingl
drwxrwxrwx@ 1 bconner  admin   32768 Aug 25 01:15 swng
BCmini-2:BlakStar bconner$

Eject the drive from the computer, cycle presets to the new font…beep. Just beep.
‘effects’ in Serial monitor simply says “Done listing effects”.
Confirm no sounds by turning on blade, shows “No sounds found: hum” etc…

So far, the computer thinks it copied the files fine, but the board thinks it’s an empty directory.

Remove SD from board, use card reader.
Take a listing in Terminal

BCmini-2:~ bconner$ cd /Volumes/PB37B/BlakStar 
BCmini-2:BlakStar bconner$ ls -al
total 128
drwxrwxrwx  1 bconner  admin  32768 Dec 31  1969 .
drwxrwxrwx@ 1 bconner  admin  32768 Nov 28 23:23 ..
BCmini-2:BlakStar bconner$

Take a look in Finder file explorer. Zero items.

Now in the past @profezzorn, you have mentioned maybe the computer is caching or otherwise somehow “holding on” to what it thought it did as a file transfer, which I’m not sure what could be done about that…but the weird thing is how the computer actually PLAYS the files from the SD card directly. So some sort of strangeness with the “table of contents” header information on the volume of the drive? or…?

Is there some inherent thing that Mass Storage uses a very basic drive mount-unmount system that it’s not playing nicely?

Weirdest thing is that this does not happen all the time at all. Usually it’s fine, but here today I can recreate this over and over.

In the meanwhile, I’ll test on another board and SD with another ProffieOS etc…

NoSloppy · November 29, 2021, 6:55am

TL;DR summary:
Using MacOS Finder and Mass Storage to copy font files to SD card on 3.7 board results in an empty directory as far as the board can see, yet computer OS shows files intact.
Using a card reader, seeing the SD as the board sees, the directory is empty.

Same process on a Proffieboard v1.5 no issues.
Same process on a Proffieboard v2.2 no issues.
It seems to maybe be a 3.7 board issue?

This post is just to go on record of what I did.

Card reader use: When trying to “move to trash” the “empty” BlakStar directory above where it shows no items in Finder, I get a pop-up Finder error “The operation can’t be completed because an unexpected error occurred (error code -1407).”

I know I can do rmdir to kill it successfully at that point. But first let’s try Finder deletion via Mass Storage USB connection.
I put the SD back into the board. I can still see the files, play them, and deleting the folder with the GUI is no issue.

When going back to card reader, the now deleted folders show as still there, albeit with a funky nondescript icon

Reformatting the SD no fix for this transfer-via-USB issue
Different SD same results.

NoSloppy · November 29, 2021, 7:20am

Lastly, no difference using rsync instead of GUI Finder drag-drop copy (likely the same thing under the hood).
Still creates an “empty” directory as far as the board is concerned, yet the computer OS shows the files are fully there and intact, and working.
Card reader proves what the board says…no files there, just the folder.

Not fun. I’m rebooting the computer

NoSloppy · November 29, 2021, 7:44am

Same using different Mac.
edit and now other folders are affected, losing their identities as directories, non descript file icons. non removable.

NoSloppy · November 29, 2021, 9:04am

Just triple confirmed not an issue on a V2.2 Proffieboard.

profezzorn · November 29, 2021, 9:13am

This doesn’t guarantee anything. The computer will cache the write and it will be able to play it from the cache at the same time as it is writing it to the card.
However, when you are ejecting the drive, it should sync everything. I’m not sure how mac actually tells you that it’s safe to eject though. There can potentially be several minutes of waiting required.

If this is only happening on 3.7, does it help if you select “low speed” in Arduino → Tools → SD ?

NoSloppy · November 29, 2021, 9:27am

That is exactly what I was testing next as you replied.
SDIO ( Default Speed ) does seem to allow this to work fine!
Copied a font over USB, ejected from computer when done, and board played it just fine.
Card reader use shows the files as well just fine and normal.

I thought the speed setting was just for read/write while the board is running ProffieOS stuff, not USB connection related?
Although apparently it’s influential in SOME way.
Cool.
So now what?

The mounted drive disappears from the system. (on desktop, in window sidebar etc…)

NoSloppy · November 29, 2021, 9:32am

sdtest both settings:

SDIO (Default Speed)

04:28:40.186 -> Time to open files: Average time: 1936.53 us
04:28:40.186 ->                              .                                                                      
04:28:40.186 ->                              :                                                                      
04:28:40.186 ->                              :                                                                      
04:28:40.186 ->                              :                                                                      
04:28:40.186 ->                              :                                                                      
04:28:40.186 ->                              :                                                                      
04:28:40.186 ->                 .        .   :   .                                                                  
04:28:40.186 ->                 :   .    :   :   :                                                                 .
04:28:40.186 ->                .: : : :  ::  :.  :..                                   :       .                   :
04:28:40.186 ->             .  :: : ::: :::  ::  ::: :                             :   :   :  .:                   :
04:28:40.186 ->             :  :: ::::: :::  ::: ::: :                             : : :  :: :::                   :
04:28:40.186 -> x100us              1                   2                   3                   4                   5
04:28:40.186 -> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
04:28:40.186 -> Time to read blocks: Average speed: 2598.91 kb/s, 29.47 simultaneous audio streams.
04:28:40.186 ->    .                                                                                                
04:28:40.186 ->    :                                                                                                
04:28:40.186 ->    :                                                                                                
04:28:40.186 ->    :                                                                                                
04:28:40.186 ->    :                                                                                                
04:28:40.186 ->    :                                                                                                
04:28:40.186 ->    :.                                                                                               
04:28:40.186 ->    ::                                                                                               
04:28:40.186 ->    ::                                                                                               
04:28:40.186 ->    ::                                                                                               
04:28:40.186 ->    ::.....                                                                                          
04:28:40.186 -> x100us              1                   2                   3                   4                   5
04:28:40.186 -> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
04:28:40.186 -> Unmounting SD Card.

.
.
High Speed

04:31:38.207 -> Time to open files: Average time: 1775.44 us
04:31:38.207 ->                            .                                                                        
04:31:38.207 ->                            :                                                                        
04:31:38.207 ->                            :                                                                        
04:31:38.240 ->                        :   :                                                                        
04:31:38.240 ->                        :   :                                                                        
04:31:38.240 ->                     :  :   :                                           .                            
04:31:38.240 ->                ..   :  :   :  .                                        :                           .
04:31:38.240 ->                ::   :  :   :  : ::                                     :                           :
04:31:38.240 ->               .::  .:  :: .: .:.::                           :  .  .   :                           :
04:31:38.240 ->            .  :::..::  :: ::.:::::                           :  :. :   :                           :
04:31:38.240 ->            : :::::::::::: :::::::::                          :  :: ::  :                           :
04:31:38.240 -> x100us              1                   2                   3                   4                   5
04:31:38.240 -> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
04:31:38.240 -> Time to read blocks: Average speed: 2846.17 kb/s, 32.27 simultaneous audio streams.
04:31:38.240 ->    .                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->    :                                                                                                
04:31:38.240 ->   :::.. . .                                                                                         
04:31:38.240 -> x100us              1                   2                   3                   4                   5
04:31:38.240 -> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
04:31:38.240 -> Battery voltage: 4.22
04:31:38.240 -> Unmounting SD Card.

MegtoothSith · November 29, 2021, 1:49pm

Ill try to reflash when I get home and see if this is potentially interfering with web usb on my 3.7.

profezzorn · November 29, 2021, 7:17pm

Well, if it turns out to be a widespread problem, I will lock out the high speed setting.
“Default speed” is already 3x faster than the V2 board, and high speed is twice as fast again, although the “high speed” seems to make very little difference in sdtest.

I do wonder what’s causing it though, because it doesn’t sound like a random communication problem. Maybe it’s a timing issue. Like maybe ProffieOS needs to wait a little bit after writing to the SD card before telling the SD card to go idle or something?

NoSloppy · November 29, 2021, 11:14pm

Let me try with a windows machine