Rescuing Sinology Hard Drive

I owned Synology NAS for over a decade. Love their consumer products and overall they last long time. I have a couple of DS214se that they worked just fine for many years. 
One day unexpectedly they stop responding over the network and I experienced something that I didn’t even know was a “thing” before I started my Google quest in search of reasons.

The blinking blue light!

image
Below there’s my story of what I did to recover my data and what I learned in the process. Apply at your risk and dismay because when it comes to data it’s like the love affair. It’s a mess and varies from couple to couple. 
Here is what I went through:
  1. What’s going on with the network today
  2. Why copying the ISO file from my NAS is so weird today
  3. Oh shit. Moment
  4. Power on/off cycles and starring at blinking lights pretending that I will figure out what they mean by guessing and then talking to myself
  5. Shutdown power and open the Synology just like when you open the hood of a car thinking that you will see something so obvious that you will be able to fix. You never do!
  6. Scouting the Internet and facepalming while still wondering why me???!!
It’s time. I got to fix this!
Synology customer rep had several key data points that I am sharing below.
[Physical Checks]
- Ensure power cable / adapter is connected correctly (disconnect & reconnect if needed).
- HDDs are connected correctly to the NAS enclosure properly.

[Hardware Test]
- Turn off the unit and remove all hard drives (noting their original locations).
- Turn on the unit (with all hard drives removed).
- The unit will then begin a hardware test which may take up to 10 minutes to complete.

If the test is successful:
- The blue power LED will remain solid/lit.
If the test is unsuccessful:
- The blue power LED will continue to flash.

If this is the case we may need to arrange for the unit to be returned for testing.

Please provide the results including full LED details (status, LAN, HDD, Power)

If the hardware test passes successfully with drives disconnected but not with the drives connected, this strongly indicates a problem with one or more of your HDDs.

I would recommend running Extended tests on these drives using manufacturer test tools.

This will isolate any issues and possible hardware failure with your HDDs.

You will need to power off your NAS and connect the drives directly to your computer (either by USB enclosure or via SATA cable).

[Samsung/Seagate HDD: Seatools]
http://www.seagate.com/gb/en/support/by … downloads/

[Western Digital HDD: Lifeguard]
http://support.wdc.com/product/download.asp?level1=6
http://wdc.custhelp.com/app/answers/det … or-windows

[Hitachi HDD: Drive Fitness Test]
https://www.hgst.com/support/hard-drive … /downloads

If you have any further questions, please don’t hesitate to get in touch.
After several zillions of checks and looking at a ton of posts online I figure out the following:
  • Likely my issue with one of the devices was hardware related to the Synology station (7 years running no stop)
  • The other device had a corrupted DSM likely due to a botched self-update
Here is what I did to recover my data and devices:
Open the Synology, unscrewed and removed the HDs, Mark the order the devices are placed inside. You will need to put them back in the same slot you removed them in the first place.
You need a device that let you hook the disks to your computer. I used UGreen and I absolutely love the product. Cheap to buy and does the job.
macOS is going to tell you that the disk is damaged and offers to initialize. Do that if you want to waste all your data and life :-) otherwise just hit the “ignore” button.
macOS can’t read the file system used by Synology, you will need a Linux machine. I resisted at the idea of creating a Virtual Machine just for that. So I first tried ext4Fuse and that didn’t work. Then I tried extFS for Mac by Paragon. A very well made piece of software, simple and blazing fast. Unfortunately, for some reason, it was unable to see everything for some reasons. It could have been easily user error though.
Since Synology offers as a means to fix/investigate a solution using Ubuntu I decided to bite the bullet and install a VM with Ubuntu 18 LTSon it.
I installed Virtualbox by Oracle (free). Downloaded the ISO of Ubuntu. And configured my VM as shown here.
image
It is super crucial that you configure two things:
Enable USB detection in the settings based on the type of hardware you have.
image
and then install Guest Additions CD images. If you have installed the minimum version of Ubuntu like I did, you will find that Guest Additions which enables critical features for you, won’t work because GCC and other programming tools are not installed with the min version which is required for the installation to proceed. To fix that do the following:
  1. Open terminal and type sudo apt-get install make gcc linux-headers-$(uname -r)
  2. sudo usermod -a -G vboxsf YOUR-USER-NAME-HERE
  3. Now select the menu itme has shown below
  4. Answer yes to run automatically (it will pick the right file to run for you)
  5. Reboot the VM
  6. Create a shared folder so that you can dump data from your Linux VM to your local mac
The last step can be done via command line as well if you know what you are doing.
image
Once you boot your VM, at the bottom select the UGreen device or whatever is your device otherwise the system will never see it.
image
Now log in to Ubuntu and open a terminal window. Type the following:
  1. sudo -i
  2. apt-get install mdadm
  3. apt-get install lvm2
You will find that the solution from Synology won’t replicate for you because at the end of the 2nd command from the list above, that configuration UI never shows up and it is not that needed anyway.
The reason (for me at least) was because no RAID configuration was found. (mdadm.conf)
To build the right configuration I had to do the following:
Launch Disks from Ubuntu’s UI
image
That will tell you what is the name of the device the detected RAID is attached to. In my case and it appears to be fairly common across all the post that I scouted online, was: /dev/md127
Now what we need is the UUID of that disk/unit and you get that using this:
sudo mdadm –examine —scan (double dashes)
From its output, all need is the UUID anything else will confuse the system and likely make you balder.
image
if you are comfortable in using vi as editor go ahead and edit etc/mdadm/mdadm.conf file.  I use joe because vi drives me insane. It never grew on me and I don’t want to :-)
Once you are done with the editing it will look like this:
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root@doesntmatter.com

# definitions of existing MD arrays
ARRAY /dev/md0 UUID=07def056:b1d95953:d24d14e3:650a1519
The part in bold is what your array/device is going to be known for if you notice my output was /dev/md/2, using that is possible but ultimately you want to go for the least trouble path possible. 
Next, we can correctly assemble the array.
mdadm –stop /dev/md127
mdadm –assemble –force /dev/md0 /dev/sd[bcde]1
Finally, you should update your initramfs to have this work after a reboot. You will notice that such suggestion is also at the top of he mdadm.conf file.
update-initramfs -u
Now you can run what Synology recommend in their tutorial (outdated?).
sudo mdadm -Asf && vgchange -ay
Now you can open the file manager and save your files.
image
If you have configured the VM to use the shared folder, you will have something like the following: sf_Recovery (that’s how I called it)
image
If you start dragging too many folders at once to your <shared_folder> you will choke the throughput so take one folder at a time to avoid to cause turmoil on a disk that eventually is already on the cliff. Don’t forget to set your host machine to not going to sleep or you will have to deal with pending copying and all that.
If you haappen to have some folders that were encrypted you might want to check the reference list below for some other fascinating gimmicks to put in practice.
I hope this saves you time as it took me a while to put all these puzzle’s pieces together. Good luck!

References

Comments

Popular posts from this blog

Postgres on Synology

The Making of Basculo

Build an independent watch app - Part II