I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable
100T is 666 spindles when you go RAID10 with 300G drives (common size in the SAS/FC area).
If you go S-ATA then it'll fit on 200 spindles using 1T drives. I'll stick to the latter variant for now because I'm too lazy to lookup Sun's pricing for SAS spindles, for my comparison below.
So, 200 spindles amounts to roughly 15 hosts (throwing in a few spares for good measure). The whole setup will comfortably fit into one rack, including the FibreChannel machinery and other fluff that you'll likely want.
Thus from the hardware side this is trivially managable, 100T is just not a lot of data nowadays.
On the software side it's up to your creativity and mostly depends on what you actually need to store. I've seen people setup commodity postgres clusters, as well as more fancy things like HDFS, GFS or homegrown storage layers that way. And it worked.
And, regardless of the indeed relatively sane pricing of the Sun (formerly StorageTek) products, the bottomline is what makes the difference.
In figures, for a 100T SAN on the 7310 you're looking at something like $50k for one head, plus around $75k for three trays. We're in the $125k ballpark, hardware-only. And I'm being rather optimistic here: This setup actually holds only 96T and the head is maxed out (3 trays max per controller/head). That means your next upgrade will incur another $25k markup for the next head, good thing you didn't ask for 150T...
Squeezing the same amount of storage into 15 min-spec supermicro pizzaboxes I arrive at roughly $3000 per node, including spindles. A good buyer will get them cheaper. This commodity cluster sets us back only ~$45k in hardware.
That's 1/3 the price of the 7310 solution, being optimistic on the Sun and pessimistic on the commodity side.
That kind of difference makes up for a lot of development effort for the custom solution - most of which is a one time investment anyways and yields better flexibility in the long run. It's also the reason why most of the big boys don't use off-the-shelf SANs for their primary storage.
FWIW, I've built one of those SuperMicro machines, with eight drives, around the beginning of the year.
Allowing for redundancy (RAID 5 on six disks for data, and mirrored the boot drives), we ended up with 6.5TB of storage at about $3500 (that includes i7 proc and 12Gb RAM, and a fancy controller). My labor added some more on top of the $3500.
Mostly, we've been happy. It is more sensitive to heat issues than our HP G5s, and of course, generates more of that heat. If I were to go with any kind of density, I'd need a much more robust cooling solution.
Yep, that spec would be over the top for a storage node, though. A large chunk likely went into the controller and the i7 whereas in a storage-node most cheapo controllers (i.e. 2x8 is cheaper than 1x16) and whatever old xeon/core2 will do.
Wrt cooling, for actual servers (i.e. not storage nodes) we've had good success with Sun XFires, the 4100+ range. They are very well built and the markup over the SuperMicro junk is minimal (around 15% last time I checked).
Among the niceties is a nice array of hot-swap fans - something that SuperMicro still doesn't seem to deliver in their popular chassis.
I'd also consider the xfires for storage nodes if they'd take 3,5" drives, but afaik all their low-end models only have max eight 2,5" slots. That's just no worthwhile density when compared to the larger SuperMicro tins (which go up to 30x3,5" now I think).
Most of the money went to the hard drives themselves. They've come down a lot since (1.5TB had just been announced), but it was around 70% of the cost. Everything else was relatively cheap.
The 2.5 drives are good, and seem to becoming typical, because they run much cooler. We've got some G5s in the same closet, with just as many drives, and they don't run nearly as warm.
You can use Hadoop to scale to 100TB using Commodity PCs and still be manageable, much easier and cheaper than you can use Oracle to do same. The featureset could easily DWARF those available using Oracle, as you can mapreduce all the data using your own hadoop jobs, and it can hold whatever kind of data you want it to - you're not limited to a static schema and precomputed summaries. HIVE is running on commodity PCs at facebook on more than 100TB, and they prefer it to their enormous, overly expensive Oracle OLAP system. So there's your answer, for one use case.
But of course, if you're updating the data often - you wouldn't use hadoop (or if you were, you would run HBase or some such on top of it). But there are many use cases where you only write once at that scale. And in those cases, from my perspective - its much nicer to scale on commodity hardware than on big iron.
Maybe you should ask Google... they seem to be doing pretty well with commodity hardware and something's telling me that they've got better reliability, higher efficiency and lower costs. Also, unused features merely add unnecessary complexity to a system. Thanks for the info on the cheaper SAN though, I'll have to look into it.
Go take a look at Adam Leventhal's work with the Fishworks stuff.
Specifically, go check out the Sun Storage 7310.
It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors.
I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets.
Again, if you're railing against something as ubiquitous as a SAN, I'm not sure there's anything I can say to change your mind.
Not that I'm really here to change your mind.
Again, the original focus was about SQL not scaling, and you seem to be fixated on the cost of that scaling.