LWM-Linux/08 - Advanced Linux Administration/System Log Management (SystemD).md

System Log Management *JournalCTL)

1. Introduction to journalctl

journalctl is a command-line utility for querying and displaying logs from the systemd journal. The systemd journal is a centralized logging system that collects and stores logging data from various sources, including the kernel, system services, and applications.

2. Basic Usage

To view all logs:

journalctl

To follow new log entries in real-time:

journalctl -f

3. Filtering Logs

By time:

journalctl --since "2024-01-01 00:00:00"

journalctl --until "2024-01-31 23:59:59"

journalctl --since "1 hour ago"

By service unit:

journalctl -u nginx.service

journalctl -u ssh.service

By priority level:

journalctl -p err Priority levels: emerg, alert, crit, err, warning, notice, info, debug

By kernel messages:

journalctl -k

4. Output Formatting

JSON output:

journalctl -o json

Short output format:

journalctl -o short

Verbose output:

journalctl -o verbose

5. Boot-specific Logs

Current boot:

journalctl -b

Previous boot:

journalctl -b -1

6. User-specific Logs

journalctl _UID=1000

7. Disk Usage and Log Rotation

View disk usage:

journalctl --disk-usage

Rotate logs:

journalctl --rotate

Vacuum old logs:

journalctl --vacuum-time=1week

journalctl --vacuum-size=1G

8. Remote Journal Access

To access logs on a remote system: journalctl -D /path/to/journal/directory

9. Persistent Journal Storage

Edit /etc/systemd/journald.conf:

Storage=persistent

Restart journald:

sudo systemctl restart systemd-journald

10. Forwarding Logs to a Central Server

Install rsyslog:

sudo apt install rsyslog

Configure /etc/rsyslog.conf for forwarding:

*.* @@central-log-server:514

Restart rsyslog:

sudo systemctl restart rsyslog

11. Security Considerations

• Restrict access to journal files
• Use encryption for remote logging
• Regularly audit and review logs
• Implement log retention policies

12. Performance Tuning

Adjust RateLimitInterval and RateLimitBurst in /etc/systemd/journald.conf to balance between logging thoroughness and system performance.

13. Integration with Other Tools

journalctl can be combined with other tools like grep, awk, and sed for advanced log analysis:

journalctl | grep "error" | awk '{print $1, $2, $3}'

14. Scripting and Automation

You can use journalctl in shell scripts for automated log analysis and reporting.