If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.

  • $\begingroup$ For those not familiar with it, could you explain what "spark" is? $\endgroup$ – Oliver Mason Feb 26 at 9:08
  • $\begingroup$ Apache Spark is open source, general-purpose distributed computing engine used for processing and analyzing a large amount of data. Just like Hadoop MapReduce, it also works with the system to distribute data across the cluster and process the data in parallel. $\endgroup$ – Sharath Feb 29 at 3:50

Yes, it is possible to use deep learning architecture with Apache Spark now. Databricks have Spark-deep-learning which is a pipeline based in python and uses tensorflow and keras.


You can check this. There is also BigDL by Intel analytics. https://github.com/intel-analytics/BigDL

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.