The AWS API is convoluted, overwhelmingly complicated, and often yields no insight to the actual behavior taking place in the Amazon system. I can think of countless scenarios wherein my API call succeeded but nothing happened, with no reporting of the reason why anywhere. A later run of the exact same call again works. No explanation.
Now, arguably, I'm discussing the implementation of the specific API, but even on an architecture level the AWS API tries to handle absolutely every case in the most complex manner possible. This is because AWS tries to handle every case in their systems; where other providers trim things for simplicity, AWS adds as much as it can. This flexibility can be useful if you need it but to someone getting started, it is a giant wall. They've gotten much better about guiding people through setup, but the API still reflects the kitchen sink philosophy. Everybody I've ever known who loves the AWS API has only ever used boto. As an exercise, launch an instance with ephemeral and EBS storage mounted using the bare API on a clean account. You'll see what boto hides from you.
The correct approach is something like libcloud with the ability to interact with platform-specific things in a way that the library does not anticipate. However, there's a school of thought in that if you are digging yourself in to platform-specific features, your ability to move is now a significant line item of technical debt that could put you in a position where you are completely stuck and cannot hire enough people to get out.
As a former hosting employee it is immeasurably frustrating that everybody mentally defaults to AWS these days without even a second thought. All of Silicon Valley is basically wired to funnel venture capital directly to Amazon. The VC firms might as well just invest in Amazon and skip the startup risk, since inevitably most of the cash burn of a startup will go directly to AWS (because it takes a genius of reservations, flexible scaling, and other tricks to not spend six figures a month on your hosting). AWS was designed for flexible scaling, not running your 700 instances 24x7 regardless of load, but nobody operates that way. You think Amazon is going to tell you that?
With Amazon, the API is full of issues, and that's even before speaking about the service itself as well as the support; even with paid support, 12-hour ticket responses were the norm for me including in an outage situation. My personal favorite was a six-month ticket that culminated in "we don't know, so we're closing this ticket," when reporting buggy behavior with their API. There are many aspects of AWS that are dreadful, the API being only a small example, and the suggestion that we should standardize on the AWS way of doing things as an industry is very harmful. I'd contend we need a viable alternative, but Amazon has made themselves a Kitchen Sink Provider and other companies can only bite at little pieces of the pie.
Very different experience with AWS/OpenStack/running our own here. We indeed had a largish AWS bill ~50k/month. Decided to move dev/test in house into an OpenStack cluster I built with the help of Mirantis/FuelWeb. Decent experience standing up the cluster but spurts of trouble thereafter. No real win with anything API-like at all with OpenStack. Feels half-baked. Getting help is damn near impossible.
Meanwhile in AWS-land, things are humming along. Using more and more of the AWS infrastructure and removing pieces we had previously built in-house. Getting lots of mileage out of boto/Java/awscli and starting to see a real paradigm shift in how to run ephemeral servers instead of the 700 instances 24x7 approach you refer to above.
I came from running my own servers, floundered for a bit on AWS, built an OpenStack cluster and strongly prefer AWS for simplicity and sheer flexibility of the API. I'm a year in running on AWS and haven't had a single issue requiring any support.
The largest benefit has come from the concepts and projects put out by Netflix. If I didn't see companies like them leading the way, I'd likely be a bigger OpenStack fanboy.
The AWS API is convoluted, overwhelmingly complicated, and often yields no insight to the actual behavior taking place in the Amazon system. I can think of countless scenarios wherein my API call succeeded but nothing happened, with no reporting of the reason why anywhere. A later run of the exact same call again works. No explanation.
Now, arguably, I'm discussing the implementation of the specific API, but even on an architecture level the AWS API tries to handle absolutely every case in the most complex manner possible. This is because AWS tries to handle every case in their systems; where other providers trim things for simplicity, AWS adds as much as it can. This flexibility can be useful if you need it but to someone getting started, it is a giant wall. They've gotten much better about guiding people through setup, but the API still reflects the kitchen sink philosophy. Everybody I've ever known who loves the AWS API has only ever used boto. As an exercise, launch an instance with ephemeral and EBS storage mounted using the bare API on a clean account. You'll see what boto hides from you.
The correct approach is something like libcloud with the ability to interact with platform-specific things in a way that the library does not anticipate. However, there's a school of thought in that if you are digging yourself in to platform-specific features, your ability to move is now a significant line item of technical debt that could put you in a position where you are completely stuck and cannot hire enough people to get out.
As a former hosting employee it is immeasurably frustrating that everybody mentally defaults to AWS these days without even a second thought. All of Silicon Valley is basically wired to funnel venture capital directly to Amazon. The VC firms might as well just invest in Amazon and skip the startup risk, since inevitably most of the cash burn of a startup will go directly to AWS (because it takes a genius of reservations, flexible scaling, and other tricks to not spend six figures a month on your hosting). AWS was designed for flexible scaling, not running your 700 instances 24x7 regardless of load, but nobody operates that way. You think Amazon is going to tell you that?
With Amazon, the API is full of issues, and that's even before speaking about the service itself as well as the support; even with paid support, 12-hour ticket responses were the norm for me including in an outage situation. My personal favorite was a six-month ticket that culminated in "we don't know, so we're closing this ticket," when reporting buggy behavior with their API. There are many aspects of AWS that are dreadful, the API being only a small example, and the suggestion that we should standardize on the AWS way of doing things as an industry is very harmful. I'd contend we need a viable alternative, but Amazon has made themselves a Kitchen Sink Provider and other companies can only bite at little pieces of the pie.