Tuesday, October 23, 2012

Hadoop Quest 1 : ERROR : IBM BIGINSIGHTS on Cloud not accessible from web links on control panel

Using commands

1. I could not get to links of web pages of NameNode status, JobTrackerStatus etc from the web page of IBM instance of Master node . But only one of the linke "BigInsights web console" is taking me to the next web page. What should I do in this regard?

Diagnosis :
I started with Assumption : From the nature of the problem I think if I restart the instance I will be good to go. But first I needed to make sure which part of Hadoop is not working -> NameNode or Jobtracker or Tasktracker? -> All are up and running.

Following is a snapshot of IBM's Biginsight's console of showing that the NameNode is also acting as a secondary NameNode and  Jobtracker



So I started with the UserGuide : http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Shell+Commands

Info is like :

Shell Commands

Hadoop includes various shell-like commands that directly interact with HDFS and other file systems that Hadoop supports. The command bin/hdfs dfs -help lists the commands supported by Hadoop shell. Furthermore, the command bin/hdfs dfs -help command-name displays more detailed help for a command. These commands support most of the normal files system operations like copying files, changing file permissions, etc. It also supports a few HDFS specific operations like changing replication of files. For more information see File System Shell Guide.

DFSAdmin Command

The bin/hadoop dfsadmin command supports a few HDFS administration related operations. The bin/hadoop dfsadmin -help command lists all the commands currently supported. For e.g.:
  • -report : reports basic statistics of HDFS. Some of this information is also available on the NameNode front page.
  • -safemode : though usually not required, an administrator can manually enter or leave Safemode.
  • -finalizeUpgrade : removes previous backup of the cluster made during last upgrade.
  • -refreshNodes : Updates the set of hosts allowed to connect to namenode. Re-reads the config file to update values defined by dfs.hosts and dfs.host.exclude and reads the entires (hostnames) in those files. Each entry not defined in dfs.hosts but in dfs.hosts.exclude is decommissioned. Each entry defined in dfs.hosts and also in dfs.host.exclude is stopped from decommissioning if it has aleady been marked for decommission. Entires not present in both the lists are decommissioned.
  • -printTopology : Print the topology of the cluster. Display a tree of racks and datanodes attached to the tracks as viewed by the NameNode.
So I used the $hadoop -report to find the report on the HDFS name and data nodes. It showed me a complete detail of all the Hadoop nodes (active + dead) -> this way I was able to diagnose one of the node "Data Node 3" having "I.P. like in xxx.xxx.xxx.37" was dead . I removed it and made another node on its place . "Pretty cool stuff right "

 Now I stopped hadoop by stop-all.sh and started all the nodes by start-all.sh but found out that hive is not started , so

  1. I started one more name node 
  2. Installed winscp
  3. Configured winscp for ssh between my laptop and newly created name node.
  4. Copied the /mnt/biginsights/opt/ibm/bidinsights/hive folder to local directory of my laptop
  5. Started one more winscp for ssh between my laptop and old name node.
  6. copied the hive files into the same location as it was earlier
  7. now go to hive directory
  8. go to /conf
  9. vi hive-site.xml
  10. in the site name -> xxx.xxx.xxx.xxx rename to ur namenode xxx.xxx.xxx.xxx 
  11. Start hadoop by start-all.sh
  12. It starts successfully.


But still the sites from the link are not opening up.

SO NOW I looked into the log files : By default the $BIGINSIGHTS_VAR has your biginsights folder so did following

  1. cd $BIGINSIGHTS_VAR/console/log
  2. vi console-wasce.log 
  3. Found following error : 
2012-10-22 15:50:48,218 ERROR [[JobServlet]] Servlet.service() for servlet JobServlet threw exception
java.lang.NullPointerException
        at com.ibm.xap.console.job.JobUtil.jobOperationExceptionHandler(Unknown Source)
        at com.ibm.xap.console.job.JobUtil.createJob(Unknown Source)
        at com.ibm.xap.console.job.JobUtil.handleJobCmd(Unknown Source)
        at com.ibm.xap.console.job.JobOperationHandler.handleJobCmd(Unknown Source)
        at com.ibm.xap.console.job.JobOperationHandler.handleCmd(Unknown Source)
        at com.ibm.xap.console.servlet.JobServlet.doGet(Unknown Source)
        at com.ibm.xap.console.servlet.JobServlet.doPost(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56)
        at org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:406)
        at org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.geronimo.tomcat.valve.ThreadCleanerValve.invoke(ThreadCleanerValve.java:40)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:736)


This is the root cause of all problems. Lets figure out whats happening in the connectionhandler.process...

I searched a lot for the jobUtil file but could not find it. So Ultimately I looked into two of the folders under /mnt/BI/opt/ibm/BI/ -> home directory for installation of hadoop BI is biginsights
  1. hadoop-conf -> all file pertaining to hadoop -> hadoop.sh hadoop-conf.xml etc
  2. conf -> biginsights-conf.sh -> all environment variable like $BIGINSIGHTS_VAR AND $ BIGINSIGHTS_HOME
Finally, I looked into the console folder under BIGINSIGHTS_HOME, which has a folder wascs -> contains information about the BIconsole.WAR file -> I think this might have got corrupt or something like that. 

Also I did tests and ran wordcount 2-3 times so no problem => hadoop is fine its webconsole war is corrupt. DONT KNOW WHAT TO DO NEXT....

UPDATE ON 10/23/2012

SO I finally resolved the problem. Apparently I found the following exception in $BIGINSIGHTS_VAR or /mnt/bI/var/ibm/BI/hadoop/logs/hadoop-<username>-namenode-<hostname>.log

2012-10-23 13:39:48,297 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call delete(/hadoop/mapred/system/job_201210221816_0013, true) from 170.224.161.37:36870: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /hadoop/mapred/system/job_201210221816_0013. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 23 seconds.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /hadoop/mapred/system/job_201210221816_0013. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 23 seconds.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

The solution is simple : This error occurs where some blocks were never reported in.So, I had to forcefully let the namenode leave safemode (hadoop dfsadmin -safemode leave) and then (optionally) run an fsck to delete missing files.

I only run the leave safenode command and than clicked on the console from the BIGINSIGHTS web console. Everything is working now.








Posted by at 1 comment:

Friday, October 12, 2012

Papers on Map Reduce

Following are a few papers that might interest you if you are in the field of Machine Learning Data Mining  and BIG DATA

Atbrox is startup company providing technology and services for Search and Mapreduce/Hadoop. Our background is from Google, IBM and research. Contact us if you need help with algorithms for mapreduce
This posting is the May 2010 update to the similar posting from February 2010, with 30 new papers compared to the prior posting, new ones are marked with *.
Motivation
Learn from academic literature about how the mapreduce parallel model and hadoop implementation is used to solve algorithmic problems.
Which areas do the papers cover?
    Ads Analysis
    For an example of Parallel Machine Learning with Hadoop/Mapreduce, check out ourprevious blog post.

Who wrote the above papers?
Companies: China Mobile, eBay, Google, Hewlett Packard and Intel, Microsoft, Wikipedia, Yahoo and Yandex.
Government Institutions and Universities: US National Security Agency (NSA)
, Carnegie Mellon University, TU Dresden, University of Pennsylvania, University of Central Florida, National University of Ireland, University of Missouri, University of Arizona, University of Glasgow, Berkeley University and National Tsing Hua University, University of California, Poznan University, Florida International University, Zhejiang University, Texas A&M University, University of California at Irvine, University of Illinois, Chinese Academy of Sciences, Vrije Universiteit, Engenharia University, State University of New York, Palacky University, University of Texas at Dallas
Posted by at No comments:

Thursday, October 4, 2012

Interview Questions BB 3


7.What are threads? Difference between preemptive and cooperative multithreading?
In preemptive multithreading, threads are preempted while they are in execution, by some other high priority thread before they're done with execution... in co-operative multithreading the threads can only voluntarily yield and cannot be preempted
  • A thread is a lightweight process
  • A thread is contained inside a process.
  • Multiple threads can exist within the same process and share resources such as memory, while different processes do not share these resources.
  • The threads of a process share the process's instructions (its code) and its context (the values that its variables reference at any given moment).
  • On a single processor, multithreading generally occurs by time-division multiplexing (as in multitasking): the processor switches between different threads. This context switching generally happens frequently enough that the user perceives the threads or tasks as running at the same time. 
  • On a multiprocessor (including multi-core system), the threads or tasks will actually run at the same time, with each processor or core running a particular thread or task.
  • Many modern operating systems directly support both time-sliced and multiprocessor threading with a process scheduler.
  • The kernel of an operating system allows programmers to manipulate threads via the system call interface. 
  • Some implementations are called a kernel thread, whereas a lightweight process (LWP) is a specific type of kernel thread that shares the same state and information.
  • Threads differ from traditional multitasking operating system processes in that:
    • processes are typically independent, while threads exist as subsets of a process
    • processes carry considerably more state information than threads, whereas multiple threads within a process share process state as well as memory and other resources
    • processes have separate address spaces, whereas threads share their address space
    • processes interact only through system-provided inter-process communication mechanisms
    • context switching between threads in the same process is typically faster than context switching between processes.
  •  In such a case, the programmer needs to be careful to avoid race conditions, and other non-intuitive behaviors. In order for data to be correctly manipulated, threads will often need to rendezvous in time in order to process the data in the correct order. Threads may also require mutually exclusive operations (often implemented using semaphores) in order to prevent common data from being simultaneously modified, or read while in the process of being modified. Careless use of such primitives can lead to deadlocks.
  • In a single-threaded program, if the main execution thread blocks on a long-running task, the entire application can appear to freeze. By moving such long-running tasks to a worker thread that runs concurrently with the main execution thread, it is possible for the application to remain responsive to user input while executing tasks in the background.
  • Operating systems schedule threads in one of two ways:
    1. Preemptive multitasking is generally considered the superior approach, as it allows the operating system to determine when a context switch should occur. The disadvantage to preemptive multithreading is that the system may make a context switch at an inappropriate time, causing lock convoy, priority inversion or other negative effects which may be avoided by cooperative multithreading.
    2. Cooperative multithreading, on the other hand, relies on the threads themselves to relinquish control once they are at a stopping point. This can create problems if a thread is waiting for a resource to become available.
  • Need to add more information on Lock Convoy ,Priority inversion, Deadlocks, Race Condition , Semaphoes, Mutex.

Posted by at No comments:

Tuesday, October 2, 2012

Interview Questions BB 5

11. Difference Between Java and C++?

Java: no support for multiple inheritance ; default dynamic binding; auto garbage collection; no pointer for accessing memory address; pure oo,all are inside class; exception handling
C++ : support multiple inheritance; default static binding; no auto garbage collection; pointer for accessing memory address; main function is not in a class;
Additionally, Java compiles into bytecode and can be run on any machine with the proper JVM. C++ needs computer specific compilers.
12. What is inherited from the base class in C++?
In principle, a derived class inherits every member of a base class except:

  • its constructor and its destructor
  • its operator=() members
  • its friends

Although the constructors and destructors of the base class are not inherited themselves, its default constructor (i.e., its constructor with no parameters) and its destructor are always called when a new object of a derived class is created or destroyed.

If the base class has no default constructor or you want that an overloaded constructor is called when a new derived object is created, you can specify it in each constructor definition of the derived class:

derived_constructor_name (parameters) : base_constructor_name (parameters) {...}
For example: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// constructors and derived classes
#include <iostream>
using namespace std;

class mother {
  public:
    mother ()
      { cout << "mother: no parameters\n"; }
    mother (int a)
      { cout << "mother: int parameter\n"; }
};

class daughter : public mother {
  public:
    daughter (int a)
      { cout << "daughter: int parameter\n\n"; }
};

class son : public mother {
  public:
    son (int a) : mother (a)
      { cout << "son: int parameter\n\n"; }
};

int main () {
  daughter cynthia (0);
  son daniel(0);
  
  return 0;
}
mother: no parameters
daughter: int parameter
 
mother: int parameter
son: int parameter


Notice the difference between which mother's constructor is called when a new daughter object is created and which when it is a son object. The difference is because the constructor declaration of daughter and son:

1
2
daughter (int a)          // nothing specified: call default
son (int a) : mother (a)  // constructor specified: call this

13. Whats is the difference between private, public, protected inheritance in c++

these access specifiers specifies the way the parent members are visible in its children.
In case of private inheritance, both public and protected member will become private in child class, and cannot be inherited further down.
In case of public inheritance, public members will remain as public in child class and can be inherited further down, where as the protected members will remain as protected in the child class.
In case of protected inheritance, public and protected members will become protected in the childclass.
To answer that question, I'd like to describe member's accesors first on my own words. If you already know this, goto next.
There are three accesors that I'm aware of: public, protected and private.
Let:
class Base {
    public:
        int publicMember;
    private:
        int privateMember;
    protected:
        int protectedMember;
};
  • Everything which is aware of Base, is also aware that Base contains publicMember.
  • Only the children (and their children) are aware that Base contains protectedMember.
  • No one but Base is aware of privateMember.
By "been aware", I mean "acknowledge the existence of, thus, it can be access it".
next:
The same happens with public, private and protected inheritance. Let's have a class Base and a class Child which inherits from Base.
  • Shall such inheritance be public, everything which is aware of Base and Child, is also aware that Child inherits from Base.
  • Shall such inheritance be protected, only Child (and its children), is aware that itself inherits from Base.
  • Shall such inheritance be private, no one other than Child is aware of such inheritance.
14. 

Friendship and inheritance

Friend functions

In principle, private and protected members of a class cannot be accessed from outside the same class in which they are declared. However, this rule does not affect friends.

Friends are functions or classes declared with the friend keyword.

If we want to declare an external function as friend of a class, thus allowing this function to have access to the private and protected members of this class, we do it by declaring a prototype of this external function within the class, and preceding it with the keyword friend:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// friend functions
#include <iostream>
using namespace std;

class CRectangle {
    int width, height;
  public:
    void set_values (int, int);
    int area () {return (width * height);}
    friend CRectangle duplicate (CRectangle);
};

void CRectangle::set_values (int a, int b) {
  width = a;
  height = b;
}

CRectangle duplicate (CRectangle rectparam)
{
  CRectangle rectres;
  rectres.width = rectparam.width*2;
  rectres.height = rectparam.height*2;
  return (rectres);
}

int main () {
  CRectangle rect, rectb;
  rect.set_values (2,3);
  rectb = duplicate (rect);
  cout << rectb.area();
  return 0;
}
24


The duplicate function is a friend of CRectangle. From within that function we have been able to access the members width and height of different objects of type CRectangle, which are private members. Notice that neither in the declaration of duplicate() nor in its later use in main() have we considered duplicate a member of classCRectangle. It isn't! It simply has access to its private and protected members without being a member.

The friend functions can serve, for example, to conduct operations between two different classes. Generally, the use of friend functions is out of an object-oriented programming methodology, so whenever possible it is better to use members of the same class to perform operations with them. Such as in the previous example, it would have been shorter to integrate duplicate() within the class CRectangle.

Friend classes

Just as we have the possibility to define a friend function, we can also define a class as friend of another one, granting that first class access to the protected and private members of the second one.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// friend class
#include <iostream>
using namespace std;

class CSquare;

class CRectangle {
    int width, height;
  public:
    int area ()
      {return (width * height);}
    void convert (CSquare a);
};

class CSquare {
  private:
    int side;
  public:
    void set_side (int a)
      {side=a;}
    friend class CRectangle;
};

void CRectangle::convert (CSquare a) {
  width = a.side;
  height = a.side;
}
  
int main () {
  CSquare sqr;
  CRectangle rect;
  sqr.set_side(4);
  rect.convert(sqr);
  cout << rect.area();
  return 0;
}
16


In this example, we have declared CRectangle as a friend of CSquare so that CRectangle member functions could have access to the protected and private members of CSquare, more concretely to CSquare::side, which describes the side width of the square.

You may also see something new at the beginning of the program: an empty declaration of class CSquare. This is necessary because within the declaration of CRectangle we refer to CSquare (as a parameter in convert()). The definition of CSquare is included later, so if we did not include a previous empty declaration for CSquare this class would not be visible from within the definition of CRectangle.

Consider that friendships are not corresponded if we do not explicitly specify so. In our example, CRectangle is considered as a friend class by CSquare, but CRectangle does not consider CSquare to be a friend, so CRectanglecan access the protected and private members of CSquare but not the reverse way. Of course, we could have declared also CSquare as friend of CRectangle if we wanted to.

Another property of friendships is that they are not transitive: The friend of a friend is not considered to be a friend unless explicitly specified.

Posted by at No comments:
Subscribe to: Posts (Atom)