Data storage

The place for discussion of hardware, software and mobile apps.
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sun Apr 03, 2016 5:48 am

So... Just thought I'd say some shit about data storage, specifically focusing on disks:

* Front end interface is not necessarily everything. At the consumer level it is hardly ever all that important when compared with your disk spindles, unless we're going below USB 3 speeds. Don't buy a fucking 7200 RPM 4 TB disk with a USB C front end. If you're going to get a nice front end, also invest in the back end. I'd say go with 10000 or 15000 RPMs. The difference is pretty big.

* Solid state drives are getting cheaper. They are touted as more reliable and in general that is the case. It is important to stick with drives that have had a few iterations of firmware as even a small bug in the logic can land you with major data loss. If you have the funds, go solid state, assuming your reads are not going to be mostly sequential (copying shit like big videos back and forth is usually sequential), in which case they're not so cost effective.

* I see folks putting things on one drive. If the data is not that important, cool. If the actual data is very important, organize it redundantly. At the very least, set up a basic mirror (known as "RAID 1"), and if possible use a hardware RAID controller. They sell multiple disk configurations with everything built in, including the mirroring (See this Amazon search). If not, you could make speed sacrifices and do a software RAID. Most of the operating systems allow for this (Apple actually has a badass setup; see this video for an explanation of the concept using some shitty hardware).

* Consider striping with redundancy with RAID 5. 3 drives or more work for this, and it's pretty cost effective. Drive geometry-wise it's more efficient to have 5 or 9 drives, but even if it's only three, you double the speed of one drive alone, and also allow for mirroring so that you can lose one of those drives and still be operational.

* FreeNAS is cool if the drives are sitting in one computer/server and never moving about. That computer/server also has to have a fuck ton of RAM to handle the overhead.

* Especially spindled (non-solid state) drives can have a high failure rate. Stay away from super cheap shit. Western Digital Green drives, for example, are pieces of shit with more rare earth metals in them and developing country children's exposure to carcinogens than they deserve to have. If you're curious about failure rates, don't just look at NewEgg or Amazon reviews or whatever. The FreeNAS forums are good at the consumer level. I've heard raving reviews for the Western Digital Reds, and those things have been in production for years. I work for a company where you see list prices for 300 GB SAS 15000 RPM spindled drives go around 2 grand because 1) the difference in quality is THAT big and 2) there are actually legal agreements in place for those who purchase them.

ANYHOW to toot my own horn, I work at a company that provides storage solutions in the form of arrays and software solutions, etc., for most of the world's evil corporations. I focus on performance tuning a lot as well as actual data protection.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Zansi'Vara
Dark Horse of many truths
Posts: 606
Joined: Mon Mar 21, 2016 6:25 pm
Location: USA

Postby Zansi'Vara » Sun Apr 03, 2016 6:02 am

WD Reds are server drives. I mean, they're fine, but they are designed to use in RAID configuration with other drives. I would go with a WD Black, preferably, or a blue if you don't wanna spend the coin. For Solid States, I would go Samsung right now. I need to invest in a SSD for my boot drive, I just don't have the coin myself right now. xD
Image
ImageImage
ImageImage
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sun Apr 03, 2016 6:16 am

Faaaancy with the WD Black! And SECONDED on the Samsung SSDs!!!
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Feydakin
Blissfully Oblivious
Posts: 1779
Joined: Sun Mar 20, 2016 7:48 pm
Location: Right Here
Contact:

Postby Feydakin » Sun Apr 03, 2016 3:41 pm

I have a 6 year old build with Samsung SSD and they are solid and fast still...

Also; http://www.theverge.com/2016/3/30/11331228/seagate-innov8-8tb-usb-c-drive-hard-drive-pricing-release-date
"Searchers after horror haunt strange, far places..." ~ H.P. Lovecraft, "From Beyond"Image
ImageImage
User avatar
Phara
The Glue To This Bitch!
Posts: 2006
Joined: Fri Mar 18, 2016 1:13 am

Postby Phara » Sun Apr 03, 2016 10:36 pm

//stickied for now cuz i need this
Image
ImageImage
ImageImage
ImageImage
Image

Image Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Thu Apr 07, 2016 9:02 pm

Feydakin wrote:I have a 6 year old build with Samsung SSD and they are solid and fast still...

Also; http://www.theverge.com/2016/3/30/11331228/seagate-innov8-8tb-usb-c-drive-hard-drive-pricing-release-date



Badass.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Thu Apr 07, 2016 9:03 pm

Also reserving a post space here for open source deduplication software and drivers. Get enough flash drives together, deduplicate, and you can have an army of lickety split fast OS images. I hate the RAM requirements of FreeNAS so I'll look for modules and drivers and shit that can be plugged into existing setups.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Corgimom
3 hours later...
Posts: 1031
Joined: Fri Mar 18, 2016 1:23 am

Postby Corgimom » Fri Apr 15, 2016 3:38 pm

If you are looking for duplication of many flash drives I wrote a little routine that lets me plug 7 flash drives into a power assisted USB external port hit return and run file duplication on all of the drives. I actually could do it on more but I ended up with file transfer errors on some of the flash drive. They sell duplicators for flash drives for a pretty fair amount of cash but after a lot of research nothing can shorten the time it takes to physically plugin the flash drives and remove them after duplication.
Image Image
Image
est 1953


Image
User avatar
Skywalker
I Feel Ya Sista... Not in that way
Posts: 978
Joined: Mon Mar 21, 2016 4:55 pm
Location: NC

Postby Skywalker » Fri Apr 15, 2016 6:23 pm

HDD failed in my mom's old 5+-year-old HP Touchsmart PC. I bought a cheap ass WD Green 1TB HDD as a replacement. Why didn't I go for a better drive, simple it is an all in one. It is impossible to repair without taking the whole damn thing apart.

I have a Dell tower that is in need of a new or used power supply cause the last one went out. I still have yet to have the funds to fix it, when I do have the funds I buy something else like a dumb ass.
Image
Iroquois
the unproven
Posts: 20
Joined: Wed Apr 27, 2016 10:40 am

Postby Iroquois » Fri Apr 29, 2016 7:15 pm

Was this post aimed at normal consumers or businesses because as you've said 10k or 15k hdds are uber expensive and if you have the cash you're better off buying tons of ssds. For data storage ie movies and so on 7200 or even 5400 wd greens are more than adequate tbh. I think most people who are modern pcs have ssds for the os drive. I also really wouldn't put very important stuff on a raid I understand why people do but the raid controller or the nas box is the single point of failure for you. I'f its really important stick it on a usb that you keep safe or put on some sort of cloud storage or do both. I wasn't aware that greens were so bad environmentally; they shouldn't be called green.
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sat Apr 30, 2016 2:59 am

Excellent note on scope! This was mainly directed at the home server crowd, in situations where dedicated storage makes sense (and there are many situations in which this would be the case). As far as personal computers go, this is less applicable.

Well, you can put together a pretty good RAID-protected setup at the consumer level, and you'd be using software RAID preferably given its flexibility, so it is less dependent on specific hardware. If you want to ride the border between consumer and professional levels, 10K and 15K hard drives can be acquired, but agreed, 7.2K RPM is fine (assuming you aren't spending silly money on front end interfaces that are way more powerful than the disks on the back end).

As for the WD greens, they are well known to be extremely fault-prone, and spinning generally at 5400 RPM, the stripe sizes you'd have to make to get any kind of performance advantage would be so big that energy savings wouldn't be that great. The "green" label is limited to power consumption (though drive spin-down is a pretty common feature), but if you're replacing disks more often, all those rare-earth metals are going places you don't want them to be.

It's like a Prius: Sure, you're not consuming as much combustible, and as far as a quality product and consumer experience goes, there is a HUGE advantage to the hybrid approach, but those batteries are brutal environmentally speaking, and far more energy has gone into their manufacture than what could be made up for in the fuel efficiency of the lifetime of the vehicle.

When it comes to SSDs, I'm a huge fan, too, just that it's important to be discriminatory at the consumer level given inconsistent means and processes of manufacture even though quality and price are going up and down respectively. It's also important not to rely on them completely if non-stop writes are going to be happening since they can only do a limited number of writes.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Corgimom
3 hours later...
Posts: 1031
Joined: Fri Mar 18, 2016 1:23 am

Postby Corgimom » Sun May 01, 2016 4:45 am

About raid arrays- while it is sort of a standard practice to run a raid array that uses partitioning and virtual drives this is actually less reliable and slower than no raid because you are increasing the number of writes to your drives and writing to even raid5 uses 5x the system resources that no raid does.
Image Image
Image
est 1953


Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sun May 01, 2016 5:25 am

On the SATA or SAS interfaces there is the simultaneous I/O, but since data is split across as many stripe drives you have, so among 5 disks in RAID 5 writes can be 4x as fast (done by 4 spindles) as with just one disk.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Corgimom
3 hours later...
Posts: 1031
Joined: Fri Mar 18, 2016 1:23 am

Postby Corgimom » Sun May 01, 2016 5:57 am

I specified virtual drives.
Image Image
Image
est 1953


Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sun May 01, 2016 5:59 am

Ohhhh, I was UI. You're so right! Striping virtual drives is foolishness!
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Corgimom
3 hours later...
Posts: 1031
Joined: Fri Mar 18, 2016 1:23 am

Postby Corgimom » Sun May 01, 2016 6:02 am

And honest, a very common practice among hosting companies. Just had a screaming match with a client to get him to hear that lowest price Raid 10 on a server that won't hold 10 drives is insane is not a bargain.
Image Image
Image
est 1953


Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Sun May 01, 2016 7:08 am

I've muttered, "fuck you, ZFS..." on more than a few occasions...
Once was lost and now am lost; was blind but now I smoke
Image
Image
Iroquois
the unproven
Posts: 20
Joined: Wed Apr 27, 2016 10:40 am

Postby Iroquois » Mon May 02, 2016 3:11 pm

cerrodepedro wrote:Excellent note on scope! This was mainly directed at the home server crowd, in situations where dedicated storage makes sense (and there are many situations in which this would be the case). As far as personal computers go, this is less applicable.

Well, you can put together a pretty good RAID-protected setup at the consumer level, and you'd be using software RAID preferably given its flexibility, so it is less dependent on specific hardware. If you want to ride the border between consumer and professional levels, 10K and 15K hard drives can be acquired, but agreed, 7.2K RPM is fine (assuming you aren't spending silly money on front end interfaces that are way more powerful than the disks on the back end).

As for the WD greens, they are well known to be extremely fault-prone, and spinning generally at 5400 RPM, the stripe sizes you'd have to make to get any kind of performance advantage would be so big that energy savings wouldn't be that great. The "green" label is limited to power consumption (though drive spin-down is a pretty common feature), but if you're replacing disks more often, all those rare-earth metals are going places you don't want them to be.

It's like a Prius: Sure, you're not consuming as much combustible, and as far as a quality product and consumer experience goes, there is a HUGE advantage to the hybrid approach, but those batteries are brutal environmentally speaking, and far more energy has gone into their manufacture than what could be made up for in the fuel efficiency of the lifetime of the vehicle.

When it comes to SSDs, I'm a huge fan, too, just that it's important to be discriminatory at the consumer level given inconsistent means and processes of manufacture even though quality and price are going up and down respectively. It's also important not to rely on them completely if non-stop writes are going to be happening since they can only do a limited number of writes.



I can't remember where I saw it but I remember seeing studies on ssds where they simulated decades of writes and many survived an insane amount of time. I'm personally not too worries about ssds.
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Postby cerrodepedro » Wed May 11, 2016 4:26 am

THANK YOU for keeping this going, everybody. Iroquois, yup:

Yes, they're pretty great
How to keep them going

I work somewhere paranoia is a commodity. Good software can also help you keep track of your writes. The main concern for me is where you're using them to supplement RAM as a sort of secondary cache tier (or in Linux speak, swap partition, Windows speak, page files, unless we're talking about an actual write cache tier).
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Psy
Neck Veins & Deep Inner Bellows...
Posts: 1993
Joined: Sat Mar 19, 2016 7:45 pm
Location: Legendary Shitposter and Junglist.

Postby Psy » Wed May 11, 2016 6:26 am

Here's some fun, I have a 5 year old (maybe 6) OCZ 2nd Gen Agility SSD. It's gotten to the age now where it's read/write speeds are about 100MB/s with no bottlenecks, but as it's a multi-drive system it more often than not runs at 50MB/s Read/Write, making it now only 10% the speed of the original day of purchase. Dammn these old SSD controllers were naff in the long run. So yeah Intel SSD's, Samsungs and a few Corsairs are the top choices.
Image
User avatar
Feydakin
Blissfully Oblivious
Posts: 1779
Joined: Sun Mar 20, 2016 7:48 pm
Location: Right Here
Contact:

Re: Data storage

Postby Feydakin » Mon Nov 28, 2016 2:40 am

I need an affordable wireless external HDD solution for media storage.

Also, probably going to go with WD Reds to replace my old storage drives that shit the bed on me, but I'm probably not going to RAID them, just like their reliability rating.

Recommendations?
"Searchers after horror haunt strange, far places..." ~ H.P. Lovecraft, "From Beyond"Image
ImageImage
User avatar
merlin
ninja with training wheels
Posts: 52
Joined: Thu Jul 14, 2016 12:49 am

Re: Data storage

Postby merlin » Fri Jan 06, 2017 7:20 pm

leftie wrote:Swapped out my Macbook's drive for an SSD and wow, what a difference. Havent looked back since.

Also, I have a desktop thats roughly 10 years old and is running WIn 10. When taking a look at the task manager performance tab, it seems like the disk is the weakest link (always maxing out at 100% usage). Im thinking of swapping it out for an SSD in the hope that itll breathe new life into the system


The Disk is has always been the bottleneck slowing down systems and the biggest reason it took so long to get mobile phones and other devices that are fast.

I'm not a data hoarder by any means. I think anyone that has more than 500 gig of data is crazy. That's how you get caught.
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Re: Data storage

Postby cerrodepedro » Tue Jan 24, 2017 8:57 pm

merlin wrote:
leftie wrote:Swapped out my Macbook's drive for an SSD and wow, what a difference. Havent looked back since.

Also, I have a desktop thats roughly 10 years old and is running WIn 10. When taking a look at the task manager performance tab, it seems like the disk is the weakest link (always maxing out at 100% usage). Im thinking of swapping it out for an SSD in the hope that itll breathe new life into the system


The Disk is has always been the bottleneck slowing down systems and the biggest reason it took so long to get mobile phones and other devices that are fast.

I'm not a data hoarder by any means. I think anyone that has more than 500 gig of data is crazy. That's how you get caught.


Hey merlin let's be friends. I like the way you're thinking.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
cerrodepedro
Words. I kill them.
Posts: 856
Joined: Mon Mar 21, 2016 10:21 pm
Location: Intermountain West

Re: Data storage

Postby cerrodepedro » Tue Jan 24, 2017 9:11 pm

Feydakin wrote:I need an affordable wireless external HDD solution for media storage.

Also, probably going to go with WD Reds to replace my old storage drives that shit the bed on me, but I'm probably not going to RAID them, just like their reliability rating.

Recommendations?


WD Red is rated consistently by the FreeNAS community and others as pretty darned reliable, so good choice, and despite my evangelizing, RAID presents a software overhead if you don't have a fancy and well-built/well-firmware'd hardware RAID controller. That said, your enclosure ought to be pretty beefy since jostling mechanical spindle drives about can put even the best of them with heads on platters. I think I may have misunderstood though. I'm a little brain foggy today. If you were planning on making a WD Red or a couple of them internal, and looking for a separate external drive, I'd say look for discounted SSDs with good ratings and updated firmware since in general the priority for external storage is speed and portability above but not to the exclusion of reliability, unless of course you plan to overwrite data a lot since solid states have limited writes.

Additionally, anybody looking to do anything that requires high performance, no matter what, should also invest in as much RAM as is reasonably possible. Even at the enterprise level (ESPECIALLY at the enterprise level), abundance of write cache is the fastest way to do really cool shit with storage, especially things like deduplicating an operating system tens of thousands of times or doing fancy maths.

Google ("best hard drive enclosures" with results limited to things published within the last month) actually gave me a cool thing with a respectable level of attention to detail: The 11 best USB 3.0 external hard drive enclosures.
Once was lost and now am lost; was blind but now I smoke
Image
Image
User avatar
Feydakin
Blissfully Oblivious
Posts: 1779
Joined: Sun Mar 20, 2016 7:48 pm
Location: Right Here
Contact:

Re: Data storage

Postby Feydakin » Wed Jan 25, 2017 8:26 pm

The WD Reds are going to be internal. I have 32GB of GSkill 1866 DDR3 right now. I can go up to 64GB on my MB. Everything is in a full sized tower CoolerMaster HAF case, which is well built and very sturdy. As for the external, no idea what I want to get yet. It's essentially going to be just sitting and left alone, not dragged around much if at all. Thanks for the link. ;)
"Searchers after horror haunt strange, far places..." ~ H.P. Lovecraft, "From Beyond"Image
ImageImage

Return to “Tech, Software & Apps”

Who is online

Users browsing this forum: No registered users and 1 guest