Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Do this: http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-ti...

Then at the end instead of writing to HBase you can write JDBC to do the insert.



JDBC works great if you have a large RDD that you want to persist in one go.

But if you are writing row by row you will need to implement your own batching algorithm and connection pooling to get any decent performance.


You could always use mapPartition to open one connection per partition.


Nice link. Spark jobs run Java code anyway, so why not write to JDBC with the results of processing. Makes perfect sense.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: