Seeing the wood for the trees, how to deal with Atlassian logs

logs-smaller.jpg

As an administrator, you need your logs to tell you exactly how your instance is performing. This article is aimed at Atlassian application administrators (Jira, Confluence, Bitbucket, Bamboo) that want to spend less time trawling logs, and have richer information about their instance available faster. Here we’ll rundown how you can read through your existing logs, and how you might be able to improve the quality of your existing logs, making them easier to read.

Firstly, I think it is important for me to say that nobody likes reading logs. There it is, I said it.

In this article we’ll be exploring how you can break the monotony of reading logs with two different techniques:

  1. Improving how you can read logs in their current state.
  2. Editing log outputs to make them more readable.

Where are my logs?

This article assumes that you know where your logs are, and how you get to them. In the event that you don’t, then here’s how to get over the first hurdle.

Your logs are stored in files on the same machine that your Atlassian instance resides. Inside your Atlassian home directory there should be a directory called “logs” or “log” with a collection of files that end with the extension “.log”. These are your magical log files! You can simply open them to read them.

TIP: If you have the ScriptRunner app, you can use the “View server log files” built-in script to view your log files, making this process even simpler.

So, how can we make reading our current bloated logs easier?

If you're looking at a log file, it usually means something has gone wrong, horribly horribly wrong.  Either something has blown up, or the instance is running slowly and you’re looking for more info about what caused the problem.

The important thing to remember is that logs are written by human beings. A computer simply spits log messages out when it is told to. Here’s a real life example. Think about it like going to a fast food restaurant. You probably already know your order before you even walk through the door (that’s why it’s your favourite right?). When you get to the counter, the server asks you what you’d like. They say the same thing every time you go. This is like logging. The computer, like the server, is told what to say and when to say it. Repeating it as and when necessary.

This makes reading logs incredibly hard, because in most cases, there is no consistency or commonality between messages, so they can either end up looking very random, or they are repeated so often that they kind of “disappear” from our consciousness.

So where do you start? The answer is with the common bits, each individual log entry will have something in common with other log entries, to start you need to focus in on those.

Here’s a basic Jira log example:

2018-05-11 06:26:00,168 http-nio-8080-exec-22 INFO anonymous 385x216x1 9d79dq 172.17.0.1 /secure/SetupLicense.jspa [c.a.plugin.util.WaitUntil] Plugins that have yet to be enabled: (1): [com.pyxis.greenhopper.jira], 287 seconds remaining

This looks pretty messy right? Well, it is but it is also cramming in a lot of information. It is fairly concise in terms of the information that is contained, it’s just a case of knowing which bits we can ignore and which parts we need. To identify the common bits and the bits that we need, let’s step through a normal troubleshooting process and see how we can mentally filter the log.

Step 1: Identify when the issue occurred - When did the problem occur. It’s likely that someone had a problem at a given date / time range (i.e. it went wrong at about 6am).

2018-05-11 06:26:00,168 http-nio-8080-exec-22 INFO

Highlighted above is the date and time, this is useful as it tells us the when. This is always the first thing we should look for when were doing fault analysis. It also acts as a great index! This helps us by making the file searchable. We can use our favourite text editor to find a specific point in time.

Tip: You can do a find on a given date or date and time string like “2018-05-11 06:3” if you know the problem happened at around 6:30. That way you can bypass a ton of logs! There, wasn’t that easier!

Step 2: Finding an actual error - In a log we have the concepts of “levels”. These levels tell us how severe the problem is.

2018-05-11 06:26:00,168 http-nio-8080-exec-22 INFO

Highlighted above we can see the “level” for this message. Again, like the message itself this is decided by a human being. The engineer who has built either Confluence or the app decided how sever the issue was to be reported.

When we’re troubleshooting there are two main levels you should be concerned with. These are “ERROR” and “FATAL”. There will be other messages around these but they mostly just help with what else was going on.

Step 3: Alright already! So what’s the actual problem??

We’ve been patient, and all of this is very interesting but what information actually helps us fix the issue. That’s a good question. The message is normally located at the end of the log line and, in the case of the default Jira configuration, is contained after the class that triggered the error.

[c.a.plugin.util.WaitUntil] Plugins that have yet to be enabled: (1): [com.pyxis.greenhopper.jira], 287 seconds remaining

The message from the log line is highlighted above. Hurrah! The message tells us that we are waiting for the "greenhopper" plugin to be enabled, and that, at the point the message is written we have 287 seconds remaining.

So, there we are, in three fairly easy steps we managed to find our error and either report that back to the relevant support team, or do something with it.

 

Can’t it be simpler?

The answer is, yes absolutely. Everything I’ve shown you so far comes from the default log configuration in Jira / Confluence / Bitbucket / Bamboo.

The key to mastering your logs is learning how you can take control of them. The good news is that you can change the logging configuration to remove some of the messages that you don’t need, making the log files a whole lot easier to read.

Changing your log configuration is as simple as editing a text file on the server that hosts your Atlassian application (Jira / Confluence / Bitbucket / Bamboo). Atlassian has a really nifty guide to configuring your logging here (https://confluence.atlassian.com/doc/configuring-logging-181535215.html). If that’s too much reading then don’t panic, here comes the TL;DR.

Where can I change my log configuration?

There are two places you can change your logging configuration. One is in the General Settings menu (under “Logging and Profiling”). The other is directly in the log properties file on the server that hosts your Atlassian application.

You will probably need administrative access to change either, but if you know what changes you want, the powers that be will (most likely) be more than happy to make the changes for you.

It is also recommended that you make your logging changes in your log properties file on the server. If you make them in the “Logging and Profiling” menu, they won’t be there when you restart. This means that if an administrator restarts your Atlassian application, you’ll lose all of your logging setup.

How can I change my log configuration?

Atlassian have some great instructions on how to update specific logging configurations. Unfortunately I can’t cover them all here as they differ between the applications. I have, however, provided some handy links to the Atlassian documentation, explaining how you can edit that config!

 

Great, so what can I do with my log configuration?

Now that we’ve learned how to edit our log configuration, here are some of the cool things that you can do to make your log file easier to read:

Create different log files for specific types of logging

This modification gives you the ability to syphon off the most important messages into a separate log files. By splitting logs of specific types into different folders you reduce the noise and make them a lot easier to read.

Reduce irrelevant log content

This modification makes logs a lot easier to read and saves space on your server.

As we examined above, there are elements in your log that you might not necessarily be interested in seeing (such as system user, or process ID). These don’t always help you to identify the issue and can sometimes cloud the log.

NOTE - Take care with removing log content, you can end up hindering yourself if you cut back the content too much!


Be precise with levels

This modification removes clutter from the logs so that we only see the input we care about.

Using the logging configuration, we can set the levels that we want specific parts of the system to log at. This can be useful when we have, say a component from an app that is putting out lots of “INFO” level messages to the log. We can change the log level to “ERROR” so that we only see output when an error occurs.

TIP: Decluttering the logs is a great way to make them more relevant. The only issue with this is that we might end up removing things from the logs that are relevant to someone else. If we want a more granular view, the best thing to do would be to create a separate log file.


Conclusion

Wading through piles of logs isn’t fun. Following the tips and tricks in this article can help you to get the most out of your logs, in the most efficient way possible.

0 comments

Comment

Log in or Sign up to comment

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you