I configured graylog .But my stack trace appears to be cut. Is there a way I can format the output to see the full stack trace, with the error and all the caused by clauses?

[skuppam]

Make sure that:

• You have read the contribution guidelines.
• You specify the logstash-gelf version and environment so it's obvious which version is affected
• You provide a reproducible test case (either descriptive of as JUnit test) if it's a bug or the expected behavior differs from the actual behavior.

[mrmanc]

I was just about to raise another issue for this but I’ll comment here.

I’ve noticed that only the stack trace of the root cause is printed when the stack trace is filtered. Looking at the code this appears to be intentional. I can see someone might only care about the root cause, but I wondered about whether you feel it would be worth separating the ability to filter uninteresting packages from the full stack trace from the ability to present the filtered root cause?

I would be happy to look at a solution but since it’s not trivial I wanted to check what you thought first.

[skuppam]

Yes, especially with Java stack traces like below example full thread is considered as a message.
This is what we expected to see

[2016-11-16T06:41:53.144+00:00] [pod1test_adminserver] [ERROR] [WSM-07501] [oracle.wsm.resources.enforcement] [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: nagios] [ecid: 1bc262c2-7822-40ef-bd0c-2b761671294e-0000002b,0] [APP: wls-management-services] Failure in Oracle WSM Agent processRequest, category=security_and_management, function=agent.function.service, application=wls-management-services, composite=null, modelObj=null, policy=null, policyVersion=null, assertionName=null.[[
oracle.wsm.common.sdk.WSMException: GenericFault : generic error
        at oracle.wsm.agent.WSMAgent.getPolRefPolMap(WSMAgent.java:1126)
        at oracle.wsm.agent.WSMAgent.processCommon(WSMAgent.java:1178)
        at oracle.wsm.agent.WSMAgent.processRequest(WSMAgent.java:563)
        at oracle.wsm.agent.handler.jaxrs.RESTResourceFilter.filter(RESTResourceFilter.java:250)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:281)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3436)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3402)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)

This is what we are getting in graylog

[2016-11-16T06:41:53.144+00:00] [pod1test_adminserver] [ERROR] [WSM-07501] [oracle.wsm.resources.enforcement] [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: nagios] [ecid: 1bc262c2-7822-40ef-bd0c-2b761671294e-0000002b,0] [APP: wls-management-services] Failure in Oracle WSM Agent processRequest, category=security_and_management, function=agent.function.service, application=wls-management-services, composite=null, modelObj=null, policy=null, policyVersion=null, assertionName=null.[[ 

[mp911de]

The cut message sounds like a length issue. IIRC then Graylog has some field options that can be adjusted. These issues usually don't happen by using Logstash + ElasticSearch.

[mp911de]

@mrmanc you raised an interesting issue.

I think it could make sense to keep only the trace of the caught exception, the first n exceptions (seen from the caught exception) or the last n exceptions (seen from the root cause exception). This could play nicely with filterStackTrace=1 (keep only 1 exception from the caught side)/filterStackTrace=-1(keep only the first exception from the cause side).

Does this make sense?

[skuppam]

Yes but I am not sure where to enable filterStackTrace=1 or -1 or false
FYI we installed graylog collector sidecar

[mp911de]

@skuppam filterStackTrace currently allows true and false. Sorry for confusion.

[skuppam]

Yes Please let me know where to set filterStackTrace=false

[mp911de]

Checkout the docs (specific to your logging framework), see http://logging.paluch.biz/examples/log4j-1.2.x.html for Log4j

[skuppam]

So are you saying that we have to install logstash-gelf in graylog server or is there any setting in graylog where I can set filterStackTrace=false

[skuppam]

I have come across below in graylog doc: http://docs.graylog.org/en/2.1/pages/collector.html

Content Splitter

One common problem when reading from plain text log files is to decide when a log message is complete. By default, the file input considers each line in a file to be a separate log message:

To solve this problem, the file input can be configured to use a PATTERN content splitter. It creates separate log messages based on a regular expression instead of newline characters. A configuration for the stack trace example above could look like this:

inputs {
graylog-server-logs {
type = "file"
path = "/var/log/graylog-server/server.log"
content-splitter = "PATTERN"
content-splitter-pattern = "^\d{4}-\d{2}-\d{2}T" // Make sure to escape the \ character!
}
}
So do we have to do this under Start pattern of a multiline message which is in Configure Beats Input

[mp911de]

I think I don't follow. In general, logstash-gelf is an online connector for GELF transmission that does not require logs to be written to a log file.

[mrmanc]

@skuppam reading your comments and the fact that you mentioned the Graylog Collector Sidecar it sounds like you are writing your logs to a file, then using the collector to send them to Graylog, in which case you are not using the logstash-gelf library. You can instead write your logs directly to Graylog / whatever else by adding the appender provided by logstash-gelf to your logging framework (Logback / Log4j / whatever) (using the docs mentioned before). If you write to a log file first and then use a collector to forward them, you will have horrible problems trying to work out where messages end, as you mentioned. Much better to skip the file. 👍

@mp911de let me know whether you want to continue discussion in this thread or a new issue 😃

[mp911de]

@mrmanc let's continue the StackTraceFilter changed discussion in another ticket to avoid further confusion with this ticket. Do you want to open a new one and describe your requirements/ideas?

[mp911de]

Closing this issue as it's not related to logstash-gelf.