If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.
2
-
1$\begingroup$ For those not familiar with it, could you explain what "spark" is? $\endgroup$– Oliver MasonCommented Feb 26, 2020 at 9:08
-
$\begingroup$ Apache Spark is open source, general-purpose distributed computing engine used for processing and analyzing a large amount of data. Just like Hadoop MapReduce, it also works with the system to distribute data across the cluster and process the data in parallel. $\endgroup$– SharathCommented Feb 29, 2020 at 3:50
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes, it is possible to use deep learning architecture with Apache Spark now. Databricks have Spark-deep-learning which is a pipeline based in python and uses tensorflow and keras.
https://github.com/databricks/spark-deep-learning
You can check this. There is also BigDL by Intel analytics. https://github.com/intel-analytics/BigDL
