Linux Admin - Log Management
systemd has changed the way system logging is managed for CentOS Linux. Instead of every daemon on the system placing logs into individual locations than using tools such as tail or grep as the primary way of sorting and filtering log entries, journald has brought a single point of administration to analyzing system logs.
The main components behind systemd logging are: journal, jounralctl, and journald.conf
journald is the main logging daemon and is configured by editing journald.conf while journalctl is used to analyze events logged by journald.
Events logged by journald include: kernel events, user processes, and daemon services.
Set the Correct System Time Zone
Before using journalctl, we need to make sure our system time is set to the correct time. To do this, we want to use timedatectl.
Let's check the current system time.
[root@centos rdc]# timedatectl status
Local time: Mon 2017-03-20 00:14:49 MDT
Universal time: Mon 2017-03-20 06:14:49 UTC
RTC time: Mon 2017-03-20 06:14:49
Time zone: America/Denver (MDT, -0600)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
Sun 2017-03-12 01:59:59 MST
Sun 2017-03-12 03:00:00 MDT
Next DST change: DST ends (the clock jumps one hour backwards) at
Sun 2017-11-05 01:59:59 MDT
Sun 2017-11-05 01:00:00 MST
[root@centos rdc]#
Currently, the system is correct to the local time zone. If your system is not, let's set the correct time zone. After changing the settings, CentOS will automatically calculate the time zone offset from the current time zone, adjusting the system clock right away.
Let's list all the time zones with timedatectl −
[root@centos rdc]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau
That is the contended output from timedatectl list-timezones. To find a specific local time-zone, the grep command can be used −
[root@centos rdc]# timedatectl list-timezones | grep -i "america/New_York"
America/New_York
[root@centos rdc]#
The label used by CentOS is usually Country/Region with an underscore instead of space (New_York versus "New York").
Now let's set our time zone −
[root@centos rdc]# timedatectl set-timezone "America/New_York"
[root@centos rdc]# date
Mon Mar 20 02:28:44 EDT 2017
[root@centos rdc]#
Your system clock should automatically adjust the time.
Use journalctl to Analyze Logs
Common command line switches when using journalctl −
| Switch |
Action |
| -k |
Lists only kernel messages |
| -u |
Lists by specific unit (httpd, sshd, etc...) |
| -b |
Boots the label offset |
| -o |
Logs the output format |
| -p |
Filters by log type (either name or number) |
| -F |
Fieldname or fieldnamevalue |
| --utc |
Time in UTC offset |
| --since |
Filter by timeframe |
Examine Boot Logs
First, we will examine and configure the boot logs in CentOS Linux. The first thing you will notice is that CentOS, by default, doesn't store boot logging that is persistent across reboots.
To check boot logs per reboot instance, we can issue the following command −
[root@centos rdc]# journalctl --list-boots
-4 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-3 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-2 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-1 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
0 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:07:36 MDT
[root@centos rdc]#
After rebooting the system, we can see another entry.
[root@centos rdc]# journalctl --list-boots
-5 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-4 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-3 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-2 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
-1 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:09:57 MDT
0 aa6aaf0f0f0d4fcf924e17849593d972 Mon 2017-03-20 01:10:07 MDT—Mon 2017-03-20 01:12:44 MDT
[root@centos rdc]#
Now, let's examine the last boot logging instance −
root@centos rdc]# journalctl -b -5
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Mon 2017-03-20 01:20:27 MDT. --
Mar 19 22:01:57 localhost.localdomain systemd-journal[97]: Runtime journal is using 8.0M
(max allowed 108.4M
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpu
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
Mar 19 22:01:57 localhost.localdomain kernel: Linux version 3.10.0514.6.2.el7.x86_64
(builder@kbuilder.dev.
Mar 19 22:01:57 localhost.localdomain kernel: Command line:
BOOT_IMAGE=/vmlinuz-3.10.0-514.6.2.el7.x86_64 ro
Mar 19 22:01:57 localhost.localdomain kernel: Disabled fast string operations
Mar 19 22:01:57 localhost.localdomain kernel: e820: BIOS-provided physical RAM map:
Above is the condensed output from our last boot. We could also refer back to a boot log from hours, days, weeks, months, and even years. However, by default CentOS doesn't store persistent boot logs. To enable persistently storing boot logs, we need to make a few configuration changes −
- Make central storage points for boot logs
- Give proper permissions to a new log folder
- Configure journald.conf for persistent logging
Configure Boot Location for Persistent Boot Logs
The initial place journald will want to store persistent boot logs is /var/log/journal. Since this doesn't exist by default, let's create it −
[root@centos rdc]# mkdir /var/log/journal
Now, let's give the directory proper permissions journald daemon access −
systemd-tmpfiles --create --prefix /var/log/journal
Finally, let's tell journald it should store persistent boot logs. In vim or your favorite text editor, open /etc/systemd/jounrald.conf".
# See journald.conf(5) for details.
[Journal]=Storage=peristent
The line we are concerned with is, Storage=. First remove the comment #, then change to Storage = persistent as depicted above. Save and reboot your CentOS system and take care that there should be multiple entries when running journalctl list-boots.
Note − A constantly changing machine-id like that from a VPS provider can cause journald to fail at storing persistent boot logs. There are many workarounds for such a scenario. It is best to peruse the current fixes posted to CentOS Admin forums, than follow the trusted advice from those who have found plausible VPS workarounds.
To examine a specific boot log, we simply need to get each offset using journald --list-boots the offset with the -b switch. So to check the second boot log we'd use −
journalctl -b -2
The default for -b with no boot log offset specified will always be the current boot log after the last reboot.
Analyze Logs by Log Type
Events from journald are numbered and categorized into 7 separate types −
0 - emerg :: System is unusable
1 - alert :: Action must be taken immediatly
2 - crit :: Action is advised to be taken immediatly
3 - err :: Error effecting functionality of application
4 - warning :: Usually means a common issue that can affect security or usilbity
5 - info :: logged informtation for common operations
6 - debug :: usually disabled by default to troubleshoot functionality
Hence, if we want to see all warnings the following command can be issued via journalctl −
[root@centos rdc]# journalctl -p 4
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Wed 2017-03-22 22:33:42 MDT. --
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: RSDP 00000000000f6a10 00024
(v02 PTLTD )
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: XSDT 0000000095eea65b 0005C
(v01 INTEL 440BX 06040000 VMW 01
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACP 0000000095efee73 000F4
(v04 INTEL 440BX 06040000 PTL 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: DSDT 0000000095eec749 1272A
(v01 PTLTD Custom 06040000 MSFT 03
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACS 0000000095efffc0 00040
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: BOOT 0000000095eec721 00028
(v01 PTLTD $SBFTBL$ 06040000 LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: APIC 0000000095eeb8bd 00742
(v01 PTLTD ? APIC 06040000 LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: MCFG 0000000095eeb881 0003C
(v01 PTLTD $PCITBL$ 06040000 LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: SRAT 0000000095eea757 008A8
(v02 VMWARE MEMPLUG 06040000 VMW 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: HPET 0000000095eea71f 00038
(v01 VMWARE VMW HPET 06040000 VMW 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: WAET 0000000095eea6f7 00028
(v01 VMWARE VMW WAET 06040000 VMW 00
Mar 19 22:01:57 localhost.localdomain kernel: Zone ranges:
Mar 19 22:01:57 localhost.localdomain kernel: DMA [mem 0x000010000x00ffffff]
Mar 19 22:01:57 localhost.localdomain kernel: DMA32 [mem 0x010000000xffffffff]
Mar 19 22:01:57 localhost.localdomain kernel: Normal empty
Mar 19 22:01:57 localhost.localdomain kernel: Movable zone start for each node
Mar 19 22:01:57 localhost.localdomain kernel: Early memory node ranges
Mar 19 22:01:57 localhost.localdomain kernel: node 0: [mem 0x000010000x0009dfff]
Mar 19 22:01:57 localhost.localdomain kernel: node 0: [mem 0x001000000x95edffff]
Mar 19 22:01:57 localhost.localdomain kernel: node 0: [mem 0x95f000000x95ffffff]
Mar 19 22:01:57 localhost.localdomain kernel: Built 1 zonelists in Node order,
mobility grouping on. Total pages: 60
Mar 19 22:01:57 localhost.localdomain kernel: Policy zone: DMA32
Mar 19 22:01:57 localhost.localdomain kernel: ENERGY_PERF_BIAS: Set to
'normal', was 'performance'
The above shows all warnings for the past 4 days on the system.
The new way of viewing and perusing logs with systemd does take little practice and research to become familiar with. However, with different output formats and particular notice to making all packaged daemon logs universal, it is worth embracing. journald offers great flexibility and efficiency over traditional log analysis methods.
