Jumbune
  1. Jumbune
  2. JUM-26

Cluster monitor fails after 3-4 hours of continous execution and throws an 'out of memory' error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: cluster monitoring
    • Labels:
      None

      Description

      Cluster monitoring stats fail to refresh after 3-4 hours of monitoring the cluster with the following exception


      WARN:oejs.ServletHandler:Error for /ProfilerServlet
      java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:691)
      at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.<init>(ClientCommunicatorAdmin.java:45)
      at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.<init>(RMIConnector.java:1462)
      at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:327)
      at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
      at org.jumbune.profiling.utils.JMXConnectorInstance.getJMXConnectorInstance(JMXConnectorInstance.java:48)
      at org.jumbune.profiling.utils.ProfilerJMXDump.getAllJMXStats(ProfilerJMXDump.java:199)
      at org.jumbune.profiling.utils.ProfilerStats.getJtStats(ProfilerStats.java:315)
      at org.jumbune.profiling.service.ClusterViewServiceImpl.calculateClusterWideStats(ClusterViewServiceImpl.java:528)
      at org.jumbune.profiling.service.ClusterViewServiceImpl.calculateStats(ClusterViewServiceImpl.java:357)
      at org.jumbune.profiling.service.ClusterViewServiceImpl.getNodeView(ClusterViewServiceImpl.java:320)
      at org.jumbune.web.servlet.ProfilerServlet.getNodeViewJson(ProfilerServlet.java:287)
      at org.jumbune.web.servlet.ProfilerServlet.service(ProfilerServlet.java:131)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:598)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
      at org.eclipse.jetty.server.Server.handle(Server.java:350)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
      at java.lang.Thread.run(Thread.java:722)

        Activity

        Show
        Purva Bindal added a comment - Fixed. Please verify [1] [1] . https://github.com/impetus-opensource/jumbune/commit/7202103ba8f4274bf056e64b735e42b94cb326fd
        Hide
        Mahesh Nair added a comment -

        Verified on Build No 53

        Show
        Mahesh Nair added a comment - Verified on Build No 53

          People

          • Assignee:
            Purva Bindal
            Reporter:
            Mahesh Nair
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: