I am new to big data theory, and during the past 3 days, I took an official big data course with some of the best instructors available in my country in this domain. The things was little bit obscure for me as I have an engineering background but with no knowledge in AI techniques and domains.

After getting an intro on big data and the 5Vs (Volume, Velocity, ...), we got an intro on hadoop and hadoop ecosystem tools (Hive, Pig, ...). Then a simple example on how to run MapReduce Java script on small data file.

So to make things clear with me, are hive, pig and other hadoop ecosystem tools, are tools to break up my large data files from different sources and servers into fast-readable files, by which we create new tables with our required fields to use them later on in machine learning scripts and feature extractions ?

P.S. by fast readable files I mean, using a scripting tools that normal relational database tools like SQL and oracle don't have it on huge data sets (1 terrabytes and above) to manage and get info from it as fast as possible ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.