I still don't understand database systems, how they work internally, and what problem they really solve.
It seems that new paradigms like R and RAM-only key values system are just simpler, faster and cheaper in programmer time. Loading everything in RAM and do a search seems like a huge saving of time and just works for most cases. Usually if you have more complex needs, you need to adapt your solution, and databases don't seem like a silver bullet.
Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case.
The example of how reddit store its data is pretty demonstrative that ultimately, you should not let a database system do all the work. Databases are just a file format to me, but the way it tries to work for you at a lower level will respond to basic cases, but when you increase complexity it's not relevant anymore.
Especially today when you have big data and machine learning, everyone should just learn to understand data manipulation. Not saying to teach yourself C all over again, but having a decent idea of the math of what indexing is really about. Forcing yourself to use a database because the company always used it isn't appealing to me.
It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm.
I guess companies like Google, Facebook, Twitter, Apple, Microsoft, LinkedIn, eBay who run their entire core business on NoSQL databases don't know what they are doing.
Facebook, Twitter, Google and LinkedIn all use MySQL. A lot of it. Yes, they use other datastores, but claiming they run their entire core business on NoSQL is patently wrong.
The key word there is entire. They do not use exclusively NoSQL storage technologies. They use a mix of NoSQL and SQL. There are tradeoffs to each, and if you have as many really, really great engineers as such places employ, it can make sense to deploy whatever database technology best suits each narrow storage problem in order to meet absurdly high levels of usage. For 95% of us, who have neither the resources or the needs of a business that large, using a relational database for everything is probably the right choice.
The post I was responding to seemed to be arguing that relational databases were redundant and that was the point I was reacting to. I certainly didn't intend to suggest they were the only game in town.
>I still don't understand database systems, how they work internally, and what problem they really solve.
The use a formal, mathematical abstraction (relational algebra), even if somewhat crapily implemented by particular RDBMS, to facilitate saving data, relationships between data, and queries, while at the same time assuring certain properties (e.g. ACID) and doing so in a platform/programming language neutral format and with a consistent and industry standard query interface.
NoSQLs, key-stores and the like are not newer developments -- they predate databases. They were found to be a bad fit for what we wanted in enterprise use, and RDBS caught on.
For uses with huge data (Google and such scale) a denormalised approach might be more practical for performance reasons, in which case developers and ad-hoc programs get to re-implement all the functionality and assurances of a RDBMS in an ad-hoc way on top of rawer stores. (And all this could be alleviated with a properly optimised RDBMS for such purposes the still respects relational algebra).
In a conventional enterprise setting on the other hand, a DB trumps NoSQL etc solutions any day of the week.
>Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case.
The whole idea is to free the data from being tied to a particular language, data structures and algorithms.
Back in 2000-2007 it was all about XML, and how we should store data in XML format and get them back with XQUERY, XPATH and the like. A lot of people bought into the hype and the resulting products. Then it was about JSON -- and we now have JSON stores. In 10 years, it will be something else, again ad-hoc.
Meanwhile SQL has worked for the past 3+ decades, and is based on a solid mathematical abstraction (relational algebra).
>It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm.
You'd be surprised. Or rather you're exactly right -- few "really remember the original idea of the inventor of that paradigm", and that's why we're moving in circles with ad-hoc technologies re-implemented 30+ years after they were discarded. Or why people jumped enthusiastically to Mongo to return crying back to PostgreSQL.
JSON is just as much rooted in math as SQL tables - it is `fix T = FiniteMap String T` plus a few primitive types, while SQL is `Collection (Set [String])` plus a few primitive types.
Of course, SQL has a nice efficiently-implementable algebra of set comprehensions, which may or may not be interesting for your particular use-case.
>JSON is just as much rooted in math as SQL tables - it is `fix T = FiniteMap String T` plus a few primitive types, while SQL is `Collection (Set [String])` plus a few primitive types.
It's not the tables that make SQL what it is, it's the relations.
Heck, it could even not have any types at all (a la dyamic languages or Sqlite), and relational algebra as an abstraction would still hold.
GB of RAM is fine until your database is one terabyte.
Also, RDBMS do constraint enforcement for you and that's absolutely critical when you work with any sufficiently big database (I work on a 100GB DB with a team of 6 people, changing here and there and I can't tell you how crucial the constraints are to maintain the data quality in face of bugs, misjudgment, etc.). And no, it's not for lack of testing. When you work on millions of super complex records, you always miss some corner cases (unless you're NASA and have a huge budget to test thoroughly)
It seems that new paradigms like R and RAM-only key values system are just simpler, faster and cheaper in programmer time. Loading everything in RAM and do a search seems like a huge saving of time and just works for most cases. Usually if you have more complex needs, you need to adapt your solution, and databases don't seem like a silver bullet.
Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case.
The example of how reddit store its data is pretty demonstrative that ultimately, you should not let a database system do all the work. Databases are just a file format to me, but the way it tries to work for you at a lower level will respond to basic cases, but when you increase complexity it's not relevant anymore.
Especially today when you have big data and machine learning, everyone should just learn to understand data manipulation. Not saying to teach yourself C all over again, but having a decent idea of the math of what indexing is really about. Forcing yourself to use a database because the company always used it isn't appealing to me.
It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm.