I use unix in the same way and for the same purpose as the described in the blog, but I have come to the opinion that once you get into the describe and visualize phase, it's much easier to just drop into R. Reading in the kind of file being worked on here is often as simple as
foo <-read.csv("foo.csv")
Getting summary descriptive statistics, item counts, scatter plots and histograms is often as easy as
summary(foo)
table(foo$col)
plot(foo$xcol, foo$ycol)
hist(foo$col).
I think that is lot simpler than a 4 or 5 command pipeline that can be mistake-prone to edit when you want to change column names or things like that. I still do these kinds of things in the shell sometimes, and I don't know if I can put my finger on when exactly I would drop into R vs write out a pipeline, but there IS a line somewhere...
I completely agree - while I love using the command-line for most tasks, R is ideally designed for this... and rightly so, since that's the whole point of the language!
R also supports the use of atomic operations on vectors (it automatically maps the operation over the vector), and other idioms that would be liabilities in other programming languages, but hugely beneficial in this one area.
I still haven't found a language that treats reading (and then processing) a CSV file more easily than R.
Conveniently, R should be really easy to pick up for someone familiar with working with a POSIX shell or Bash. For example, to see all variables defined in the current namespace, just type ls().
R is basically the POSIX mentality applied specifically to data processing, instead of general-purpose work.
One exception though. Piping unix command this way allow handling of _very_ large file (stream model), whereas R requires loading the whole content in memory.
That depends on the statistical moments you're calculating. For sums and means, yes. For percentiles, standard deviation, and other moments, you'll need to operate on the set as a whole, which either means re-reading the dataset or processing it in memory.
That said, I've got my own univariate statistics generator written in awk that can handle sets with millions of values readily on desktop/laptop hardware.
I'd disagree that R is trivial to pick up. I find a lot of impedance crossing tool boundaries, and much of R is written by (and for) advanced stats use, making trivial application somewhat daunting.
Which isn't to say that it's not a phenomenally powerful and capable tool. I'm just referring to its accessibility.
Every developer is familiar with Shell and Unix tools, it's trivially simple to lookup documentation and get something working really quick with unix utilities even if you are not familiar with the tool in the beginning. On the other hand the idea of learning an entirely new programming language for quick and dirty trivial stuff is a little off-putting. That being said I did not know how simple it is in R to play with data, will have to check it out.
foo <-read.csv("foo.csv")
Getting summary descriptive statistics, item counts, scatter plots and histograms is often as easy as
summary(foo)
table(foo$col)
plot(foo$xcol, foo$ycol)
hist(foo$col).
I think that is lot simpler than a 4 or 5 command pipeline that can be mistake-prone to edit when you want to change column names or things like that. I still do these kinds of things in the shell sometimes, and I don't know if I can put my finger on when exactly I would drop into R vs write out a pipeline, but there IS a line somewhere...