Showing posts with label Pig. Show all posts
Showing posts with label Pig. Show all posts
Friday, June 2, 2017
Java Experience latest Interview Quetions
1. Write the program to reverse the string without modify the special character?
2. Write the program to compare the object using comparable interface?
3. Write the program for singleton?
4. Explain concurrent packages?
5. Difference between @inject and @service annotation
6. Advantages of microservice and cassendra?
7. Write the program for executor library?
8. Whats inner join?
9. Write the program for rest controller?
10. Can we have multiple session factory in Hibernate?
11. Explain IOC container and dependency injection?
12. How many test cases you will write to test a method?
Friday, October 14, 2016
Scala vs. Groovy vs. Clojure
Scala
Scala evolved out of a
pure functional language known as Funnel and
represents a clean-room implementation of almost all Java's syntax, differing
only where a clear improvement could be made or where it would compromise the
functional nature of the language. Such differences include singleton objects
instead of static methods, and type inference.
Much of this was based
on Martin Odersky's prior work with the Pizza language. The
OO/FP integration goes far beyond mere closures and has led to the language
being described as post-functional.
Despite this, it's the closest to Java in many
ways. Mainly due to a combination of OO support and static typing, but also due
to a explicit goal in the language design that it should integrate very tightly
with Java.
Groovy
Groovy explicitly tackles two of Java's
biggest criticisms by
·
being dynamically
typed, which removes a lot of boilerplate and
·
adding closures to the
language.
It's perhaps syntactically closest to Java, not offering some
of the richer functional constructs that Clojure and Scala provide, but still
offering a definite evolutionary improvement - especially for writing
script-syle programs.
Groovy has the strongest commercial backing of
the three languages, mostly via springsource.
Clojure
Clojure is a functional language in the LISP
family, it's also dynamically typed.
Features such as STM
support give it some of the best out-of-the-box concurrency support, whereas
Scala requires a 3rd-party library such as Akka to
duplicate this.
Syntactically, it's also the furthest of the
three languages from typical Java code.
Thursday, July 7, 2016
Hadoop
Hadoop is a a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
There are four main modules in Hadoop.
- Hadoop Common: The common utilities that support the other Hadoop modules.
- Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
- Hadoop YARN: A framework for job scheduling and cluster resource management.
- Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
Before going further, Let's note that we have three different types of data.
- Structured: Structured data has strong schema and schema will be checked during write & read operation. e.g. Data in RDBMS systems like Oracle, MySQL Server etc.
- Unstructured: Data does not have any structure and it can be any form - Web server logs, E-Mail, Images etc.
- Semi-structured: Data is not strictly structured but have some structure. e.g. XML files.
Depending on type of data to be processed, we have to choose right technology.
Some more projects, which are part of Hadoop:
- HBase™: A scalable, distributed database that supports structured data storage for large tables.
- Hive™: A data warehouse infrastructure that provides data summarization and ad-hoc querying.
- Pig™: A high-level data-flow language and execution framework for parallel computation.
Hive is better than PIG in: Partitions, Server, Web interface & JDBC/ODBC support.
Some differences:
- Hive is best for structured Data & PIG is best for semi structured data
- Hive is used for reporting & PIG for programming
- Hive is used as a declarative SQL & PIG as a procedural language
- Hive supports partitions & PIG does not
- Hive can start an optional thrift based server & PIG cannot
- Hive defines tables beforehand (schema) + stores schema information in a database & PIGdoesn't have a dedicated metadata of database
- Hive does not support Avro but PIG does. EDIT: Hive supports Avro, specify the serde as org.apache.hadoop.hive.serde2.avro
- Pig also supports additional COGROUP feature for performing outer joins but hive does not. But both Hive & PIG can join, order & sort dynamically.
HBASE won't replace Map Reduce. HBase is scalable distributed database & Map Reduce is programming model for distributed processing of data. Map Reduce may act on data in HBASE in processing.
You can use HIVE/HBASE for structured/semi-structured data and process it with Hadoop Map Reduce
You can use SQOOP to import structured data from traditional RDBMS database Oracle, SQL Server etc and process it with Hadoop Map Reduce
You can use FLUME for processing Un-structured data and process with Hadoop Map Reduce
Have a look at: Hadoop Use Cases.
Hive should be used for analytical querying of data collected over a period of time. e.g Calculate trends, summarize website logs but it can't be used for real time queries.
HBase fits for real-time querying of Big Data. Facebook use it for messaging and real-time analytics.
PIG can be used to construct dataflows, run a scheduled jobs, crunch big volumes of data, aggregate/summarize it and store into relation database systems. Good for ad-hoc analysis.
Hive can be used for ad-hoc data analysis but it can't support all un-structured data formats unlike PIG.
Real time processing is needed where fast decisions need to be taken in case of Fire alarm send by sensor or fraud detection in case of banking transactions. Batch processing is needed to summarize data which can be feed into BI systems.
we used Hadoop ecosystem technologies for above applications.
Real Time Processing
Apache Storm: Stream Data processing, Rule application
HBase: Datastore for serving Realtime dashboard
Batch Processing Hadoop: Crunching huge chunk of data. 360 degrees overview or adding context to events. Interfaces or frameworks like Pig, MR, Spark, Hive, Shark help in computing. This layer needs scheduler for which Oozie is good option.
Event Handling layer
Apache Kafka was first layer to consume high velocity events from sensor. Kafka serves both Real Time and Batch analytics data flow through Linkedin connectors.
Subscribe to:
Posts (Atom)