we are looking for fluentD configuration which pushes Atlassian all logs to ELK Stack. we are using the below configuration, however, we are noticing some portion of logs is getting truncated. not sure what's wrong.
Is there anyone who is using FluentD as a log shipping agent for Atlassian Logs? Could you Please share the Format Expression, please?
########## td-agent.conf ##############
<source>
@type tail
@label @Jira_all_logs
path /data/jira-home/log/atlassian-jira.log
pos_file /data/jira-home/log/jira-log.pos
tag env.test-jira
<parse>
@type multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}-\d{4}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}-\d{4})\s+(?<thread>.*)\s(?<level>(WARN|DEBUG|INFO|ERROR|FATAL))\s+(.*)?\s*\[(?<class>[^\s]*)\]\s+(?<message>.*)/
</parse>
</source>
<label @Jira_all_logs>
<match **>
@type forward
transport tls
tls_verify_hostname true
heartbeat_type tcp
heartbeat_interval 5
<server>
host XXXXXXX
port 24224
</server>
</match>
</label>
###############################################
The below log trace is getting truncated or not being pushed to ELK. no errors reported in td-agent.log. but some other similar entries are being pushed.
2020-11-23 09:45:41,017-0800 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
___ User Directories (Ordered) ______________
DELEGATING : JIRA Delegated Authentication Directory
Implementing Class : com.atlassian.crowd.directory.DelegatedAuthenticationDirectory
Allowed Operations : CREATE_GROUP,
CREATE_ROLE,
CREATE_USER,
DELETE_GROUP,
DELETE_ROLE,
DELETE_USER,
UPDATE_GROUP,
UPDATE_GROUP_ATTRIBUTE,
UPDATE_ROLE,
UPDATE_ROLE_ATTRIBUTE,
UPDATE_USER,
UPDATE_USER_ATTRIBUTE
Encryption Type :
Active : true
Attributes :
autoAddGroups : service-desk-customers
crowd.delegated.directory.auto.create.user : true
crowd.delegated.directory.auto.update.user : false
crowd.delegated.directory.importGroups : false
---
----
