System administration is full of repetitive tasks: creating users, checking disk space, rotating logs, auditing who’s logged in, restarting misbehaving services. Each one is simple in isolation. Done manually, day after day, they consume hours and introduce human error. This guide focuses on the practical side — real scripts you can adapt and use, built around the tasks that fill a sysadmin’s week.
The Sysadmin Script Starter Template
Every script in this guide uses this foundation:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly TIMESTAMP="$(date '+%Y%m%d_%H%M%S')"
readonly LOG_FILE="/var/log/sysadmin/${SCRIPT_NAME%.sh}.log"
# Create log directory if needed
mkdir -p "$(dirname "$LOG_FILE")"
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
die() { log "ERROR: $*" >&2; exit 1; }
warn() { log "WARN: $*"; }
info() { log "INFO: $*"; }
# Ensure running as root
require_root() {
[[ $EUID -eq 0 ]] || die "This script must be run as root"
}
|
User Management Automation
Bulk User Creation from CSV
A common task when onboarding a team. Given a CSV file like:
username,fullname,group,shell
jsmith,John Smith,developers,/bin/bash
amartinez,Alice Martinez,ops,/bin/bash
blee,Bob Lee,developers,/bin/sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
|
#!/usr/bin/env bash
set -euo pipefail
# create-users.sh — Bulk create users from a CSV file
CSV_FILE="${1:?Usage: $0 <users.csv>}"
DEFAULT_PASSWORD_LENGTH=16
SKEL_DIR="/etc/skel"
log() { echo "[$(date '+%H:%M:%S')] $*"; }
generate_password() {
tr -dc 'A-Za-z0-9!@#$%^&*' < /dev/urandom | head -c "$DEFAULT_PASSWORD_LENGTH"
}
# Skip header line
tail -n +2 "$CSV_FILE" | while IFS=',' read -r username fullname group shell; do
# Strip whitespace
username="${username// /}"
group="${group// /}"
shell="${shell// /}"
# Skip blank lines
[[ -z "$username" ]] && continue
# Check if user already exists
if id "$username" &>/dev/null; then
log "SKIP: User '$username' already exists"
continue
fi
# Create group if it doesn't exist
if ! getent group "$group" &>/dev/null; then
groupadd "$group"
log "Created group: $group"
fi
# Generate a random initial password
password="$(generate_password)"
# Create the user
useradd \
--comment "$fullname" \
--gid "$group" \
--shell "$shell" \
--create-home \
--skel "$SKEL_DIR" \
"$username"
# Set password and force change on first login
echo "${username}:${password}" | chpasswd
chage --lastday 0 "$username" # Force password change on first login
log "Created user: $username ($fullname) — group: $group — temp password: $password"
done
log "Done. Send passwords to users securely (never via plaintext email)."
|
Bulk User Deactivation
For off-boarding — lock accounts, kill sessions, optionally archive home directories:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
#!/usr/bin/env bash
set -euo pipefail
# offboard-user.sh — Safely deactivate a departed user
USERNAME="${1:?Usage: $0 <username> [--archive]}"
ARCHIVE="${2:-}"
ARCHIVE_DIR="/srv/user-archives"
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "/var/log/offboard.log"; }
id "$USERNAME" &>/dev/null || { echo "User '$USERNAME' does not exist"; exit 1; }
# 1. Lock the account (prepend ! to password hash)
usermod --lock "$USERNAME"
log "Locked account: $USERNAME"
# 2. Kill all active sessions
pkill -u "$USERNAME" 2>/dev/null || true
log "Killed active sessions for: $USERNAME"
# 3. Expire the account immediately (prevents future logins even if unlocked)
chage --expiredate 1 "$USERNAME"
log "Expired account: $USERNAME"
# 4. Optionally archive home directory
HOME_DIR="$(getent passwd "$USERNAME" | cut -d: -f6)"
if [[ "$ARCHIVE" == "--archive" && -d "$HOME_DIR" ]]; then
mkdir -p "$ARCHIVE_DIR"
ARCHIVE_FILE="${ARCHIVE_DIR}/${USERNAME}_${TIMESTAMP}.tar.gz"
tar -czf "$ARCHIVE_FILE" -C "$(dirname "$HOME_DIR")" "$(basename "$HOME_DIR")"
log "Archived home directory to: $ARCHIVE_FILE"
fi
# 5. Record the action
log "Offboarding complete for: $USERNAME"
|
User Audit Report
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
#!/usr/bin/env bash
set -euo pipefail
# user-audit.sh — Report on all local user accounts
echo "=== User Account Audit — $(date '+%Y-%m-%d %H:%M') ==="
echo ""
printf "%-20s %-6s %-20s %-15s %-10s %s\n" \
"USERNAME" "UID" "FULL NAME" "LAST LOGIN" "SHELL" "STATUS"
printf "%-20s %-6s %-20s %-15s %-10s %s\n" \
"--------" "---" "---------" "----------" "-----" "------"
while IFS=: read -r user _ uid _ comment home shell; do
# Only show regular users (UID 1000+) and root
[[ "$uid" -ge 1000 || "$uid" -eq 0 ]] || continue
# Get last login
last_login=$(lastlog -u "$user" 2>/dev/null | awk 'NR==2 {
if ($2 == "Never") print "Never"
else print $4"-"$5"-"$9
}')
# Check if account is locked
status="active"
passwd_status=$(passwd -S "$user" 2>/dev/null | awk '{print $2}')
[[ "$passwd_status" == "L" || "$passwd_status" == "LK" ]] && status="LOCKED"
# Check if account is expired
expiry=$(chage -l "$user" 2>/dev/null | grep "Account expires" | cut -d: -f2 | xargs)
if [[ "$expiry" != "never" && "$expiry" != "" ]]; then
expiry_epoch=$(date -d "$expiry" +%s 2>/dev/null || echo 0)
now_epoch=$(date +%s)
[[ "$expiry_epoch" -lt "$now_epoch" ]] && status="EXPIRED"
fi
printf "%-20s %-6s %-20s %-15s %-10s %s\n" \
"$user" "$uid" "${comment%%,*}" "${last_login:-unknown}" \
"$(basename "$shell")" "$status"
done < /etc/passwd
echo ""
echo "--- Sudoers ---"
grep -E '^[^#]' /etc/sudoers 2>/dev/null | grep -v '^Defaults' || true
ls /etc/sudoers.d/ 2>/dev/null | while read -r f; do
echo " sudoers.d/$f:"
grep -E '^[^#%]' "/etc/sudoers.d/$f" 2>/dev/null | head -5 || true
done
echo ""
echo "--- Currently Logged In ---"
who
|
Disk Space Monitoring and Cleanup
Disk Usage Alert
Run this from cron. It emails when any filesystem exceeds a threshold:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
#!/usr/bin/env bash
set -euo pipefail
# disk-alert.sh — Alert when disk usage exceeds threshold
THRESHOLD=80 # Percent
ALERT_EMAIL="ops@example.com"
HOSTNAME="$(hostname -f)"
ALERTS=""
while read -r filesystem size used avail pct mountpoint; do
# Strip % sign from pct
usage="${pct%%%}"
if [[ "$usage" -gt "$THRESHOLD" ]]; then
ALERTS+=" ${mountpoint}: ${pct} used (${used} of ${size})\n"
fi
done < <(df -h --output=source,size,used,avail,pcent,target | tail -n +2 | grep -v tmpfs)
if [[ -n "$ALERTS" ]]; then
{
echo "Subject: [DISK ALERT] $HOSTNAME filesystems above ${THRESHOLD}%"
echo ""
echo "The following filesystems on $HOSTNAME are above ${THRESHOLD}% capacity:"
echo ""
printf "%b" "$ALERTS"
echo ""
echo "--- Full df output ---"
df -h
} | sendmail "$ALERT_EMAIL"
echo "Alert sent for: $ALERTS"
fi
|
Cron entry:
*/15 * * * * /usr/local/bin/disk-alert.sh
Find and Clean Large Files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
#!/usr/bin/env bash
set -euo pipefail
# find-large-files.sh — Find top disk consumers and optionally clean old logs
SCAN_DIR="${1:-/}"
MIN_SIZE="${2:-100M}"
DRY_RUN="${3:-true}"
echo "=== Large Files in $SCAN_DIR (>= $MIN_SIZE) ==="
find "$SCAN_DIR" \
-xdev \
-type f \
-size +"$MIN_SIZE" \
-printf '%s\t%p\n' 2>/dev/null \
| sort -rn \
| head -30 \
| awk '{printf "%8.1f MB\t%s\n", $1/1048576, $2}'
echo ""
echo "=== Old Log Files (>30 days, in /var/log) ==="
find /var/log -type f \( -name "*.log.*" -o -name "*.gz" \) -mtime +30 -printf '%p\n'
if [[ "$DRY_RUN" != "true" ]]; then
echo ""
echo "Cleaning old compressed logs..."
find /var/log -type f \( -name "*.log.*" -o -name "*.gz" \) -mtime +30 -delete
echo "Cleaning old core dumps..."
find /var/crash /var/core -type f -mtime +7 -delete 2>/dev/null || true
echo "Cleaning old temp files..."
find /tmp -type f -mtime +3 -delete 2>/dev/null || true
echo "Done."
fi
|
Inode Exhaustion Detection
A filesystem can be 10% full by size but 100% full by inodes — and refuse to create new files:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
#!/usr/bin/env bash
# inode-check.sh — Alert on inode exhaustion
THRESHOLD=80
echo "=== Inode Usage Report ==="
df -i | awk -v thresh="$THRESHOLD" '
NR==1 { print; next }
/^\/dev/ {
gsub(/%/, "", $5)
if ($5+0 > thresh)
printf "WARNING: %-20s %s inodes used\n", $6, $5"%"
else
printf "OK: %-20s %s inodes used\n", $6, $5"%"
}'
|
Log Management and Analysis
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
#!/usr/bin/env bash
set -euo pipefail
# log-summary.sh — Summarize errors across system logs
SINCE="${1:-today}" # e.g. "today", "1 hour ago", "2026-03-24"
echo "=== System Error Summary since: $SINCE ==="
echo ""
# systemd journal errors
echo "--- Journal Errors ---"
journalctl --since="$SINCE" -p err --no-pager -o short-iso 2>/dev/null \
| tail -50 \
| grep -v '^-- ' \
|| echo " (none)"
echo ""
echo "--- Auth Failures (SSH brute force) ---"
journalctl --since="$SINCE" -u sshd --no-pager 2>/dev/null \
| grep "Failed password" \
| awk '{print $(NF-3)}' \
| sort | uniq -c | sort -rn \
| head -20 \
| awk '{printf " %5d failed attempts from: %s\n", $1, $2}'
echo ""
echo "--- Top 10 SSH Source IPs (successful logins) ---"
journalctl --since="$SINCE" -u sshd --no-pager 2>/dev/null \
| grep "Accepted" \
| awk '{print $(NF-3), $(NF-5)}' \
| sort | uniq -c | sort -rn \
| head -10 \
| awk '{printf " %5d logins: user=%-15s from=%s\n", $1, $2, $3}'
echo ""
echo "--- OOM Killer Events ---"
journalctl --since="$SINCE" -k --no-pager 2>/dev/null \
| grep -i "oom\|out of memory\|killed process" \
|| echo " (none)"
echo ""
echo "--- Disk I/O Errors ---"
journalctl --since="$SINCE" -k --no-pager 2>/dev/null \
| grep -iE "I/O error|SCSI error|hard resetting" \
|| echo " (none)"
|
Log Rotation Script (Without logrotate)
For custom application logs not covered by the system logrotate config:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
#!/usr/bin/env bash
set -euo pipefail
# rotate-log.sh — Rotate a single log file with retention
LOG_FILE="${1:?Usage: $0 <logfile> [keep_count]}"
KEEP="${2:-7}"
[[ -f "$LOG_FILE" ]] || { echo "Log file not found: $LOG_FILE"; exit 0; }
BASE="${LOG_FILE}"
# Rotate: N-1 → N, N-2 → N-1, ..., 0 → 1
for (( i=KEEP-1; i>=1; i-- )); do
src="${BASE}.${i}.gz"
dst="${BASE}.$((i+1)).gz"
[[ -f "$src" ]] && mv "$src" "$dst"
done
# Compress current .1 if it exists
[[ -f "${BASE}.1" ]] && gzip -f "${BASE}.1"
# Move current log to .1
mv "$LOG_FILE" "${BASE}.1"
# Create fresh log file with same ownership/permissions
touch "$LOG_FILE"
chown --reference="${BASE}.1" "$LOG_FILE"
chmod --reference="${BASE}.1" "$LOG_FILE"
# Send SIGHUP to reload log file handle (for daemons that support it)
# pkill -HUP myapp
# Remove old rotations beyond KEEP
for (( i=KEEP+1; i<=KEEP+5; i++ )); do
rm -f "${BASE}.${i}.gz"
done
echo "Rotated: $LOG_FILE (keeping $KEEP)"
|
Service Health Monitoring
Multi-Service Health Check
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
#!/usr/bin/env bash
set -euo pipefail
# service-health.sh — Check and auto-restart critical services
SERVICES=(nginx postgresql redis-server docker)
ALERT_EMAIL="ops@example.com"
HOSTNAME="$(hostname -f)"
RESTART_LOG="/var/log/service-restarts.log"
restarted=()
failed=()
for service in "${SERVICES[@]}"; do
if ! systemctl is-active --quiet "$service"; then
echo "[$(date '+%H:%M:%S')] $service is DOWN — attempting restart"
if systemctl restart "$service" 2>/dev/null; then
sleep 3
if systemctl is-active --quiet "$service"; then
restarted+=("$service")
echo "[$(date '+%H:%M:%S')] $service restarted successfully" \
| tee -a "$RESTART_LOG"
else
failed+=("$service")
echo "[$(date '+%H:%M:%S')] $service FAILED to restart" \
| tee -a "$RESTART_LOG"
fi
else
failed+=("$service")
fi
fi
done
# Send alert if anything failed permanently
if [[ ${#failed[@]} -gt 0 ]]; then
{
echo "Subject: [SERVICE ALERT] Failed services on $HOSTNAME"
echo ""
echo "The following services could not be restarted on $HOSTNAME:"
printf ' - %s\n' "${failed[@]}"
echo ""
if [[ ${#restarted[@]} -gt 0 ]]; then
echo "These services were auto-restarted successfully:"
printf ' - %s\n' "${restarted[@]}"
fi
echo ""
echo "--- systemctl status ---"
for svc in "${failed[@]}"; do
systemctl status "$svc" --no-pager -l 2>&1 | head -20
echo ""
done
} | sendmail "$ALERT_EMAIL"
fi
|
HTTP Endpoint Health Check
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
#!/usr/bin/env bash
set -euo pipefail
# http-health.sh — Check HTTP endpoints and alert on failures
declare -A ENDPOINTS=(
["main site"]="https://example.com/health"
["api"]="https://api.example.com/ping"
["admin"]="https://admin.example.com/"
)
TIMEOUT=10
ALERT_EMAIL="ops@example.com"
STATUS_FILE="/var/run/http-health-status"
failures=()
for name in "${!ENDPOINTS[@]}"; do
url="${ENDPOINTS[$name]}"
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
--connect-timeout "$TIMEOUT" \
--max-time "$((TIMEOUT * 2))" \
"$url" 2>/dev/null || echo "000")
if [[ "$http_code" =~ ^2 ]]; then
echo "OK ($http_code): $name — $url"
# Clear any prior failure state for this endpoint
rm -f "${STATUS_FILE}.${name// /_}"
else
echo "FAIL ($http_code): $name — $url"
failure_file="${STATUS_FILE}.${name// /_}"
# Only alert on the first failure (don't spam on repeated checks)
if [[ ! -f "$failure_file" ]]; then
touch "$failure_file"
failures+=("$name ($url) → HTTP $http_code")
fi
fi
done
if [[ ${#failures[@]} -gt 0 ]]; then
{
echo "Subject: [HTTP ALERT] Endpoint failures on $(hostname)"
echo ""
printf ' FAIL: %s\n' "${failures[@]}"
} | sendmail "$ALERT_EMAIL"
fi
|
Process Memory Leak Detection
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
#!/usr/bin/env bash
# memory-monitor.sh — Alert when a process exceeds a memory threshold
PROCESS_NAME="${1:?Usage: $0 <process_name> [threshold_mb]}"
THRESHOLD_MB="${2:-500}"
ALERT_EMAIL="ops@example.com"
# Get all PIDs matching the process name
while read -r pid rss vsz comm; do
rss_mb=$(( rss / 1024 ))
if [[ "$rss_mb" -gt "$THRESHOLD_MB" ]]; then
echo "WARNING: $comm (PID $pid) using ${rss_mb}MB RSS (threshold: ${THRESHOLD_MB}MB)"
# Capture process details
details=$(ps -p "$pid" -o pid,ppid,user,pcpu,pmem,rss,vsz,etime,cmd --no-headers 2>/dev/null)
open_fds=$(ls /proc/"$pid"/fd 2>/dev/null | wc -l)
{
echo "Subject: [MEMORY ALERT] $comm PID $pid using ${rss_mb}MB"
echo ""
echo "Process $comm (PID $pid) has exceeded ${THRESHOLD_MB}MB RSS on $(hostname)"
echo ""
echo "Current RSS: ${rss_mb}MB"
echo "Open file descriptors: $open_fds"
echo ""
echo "Process details:"
echo "$details"
echo ""
echo "Top memory users:"
ps aux --sort=-%mem | head -10
} | sendmail "$ALERT_EMAIL"
fi
done < <(ps -C "$PROCESS_NAME" -o pid,rss,vsz,comm --no-headers 2>/dev/null)
|
System Maintenance Automation
Automated Security Updates
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
|
#!/usr/bin/env bash
set -euo pipefail
# security-updates.sh — Apply security patches automatically
LOG="/var/log/auto-security-updates.log"
REBOOT_REQUIRED_FILE="/var/run/reboot-required"
ALERT_EMAIL="ops@example.com"
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }
log "Starting security update check"
if command -v apt-get &>/dev/null; then
# Debian/Ubuntu
apt-get update -qq 2>&1 | tee -a "$LOG"
UPDATES=$(apt-get --just-print upgrade 2>/dev/null \
| grep -c "^Inst" || echo 0)
SECURITY_UPDATES=$(apt-get --just-print upgrade 2>/dev/null \
| grep "^Inst.*security" | wc -l)
if [[ "$SECURITY_UPDATES" -gt 0 ]]; then
log "Applying $SECURITY_UPDATES security updates"
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confold" \
2>&1 | tee -a "$LOG"
else
log "No security updates available"
fi
elif command -v dnf &>/dev/null; then
# RHEL/Fedora
SECURITY_UPDATES=$(dnf check-update --security -q 2>/dev/null | wc -l)
if [[ "$SECURITY_UPDATES" -gt 0 ]]; then
log "Applying $SECURITY_UPDATES security updates"
dnf upgrade --security -y 2>&1 | tee -a "$LOG"
else
log "No security updates available"
fi
fi
# Check if reboot is required
if [[ -f "$REBOOT_REQUIRED_FILE" ]]; then
log "REBOOT REQUIRED after updates"
{
echo "Subject: [MAINTENANCE] Reboot required on $(hostname)"
echo ""
echo "Security updates were applied to $(hostname) and a reboot is required."
echo ""
cat "$LOG" | tail -30
} | sendmail "$ALERT_EMAIL"
fi
log "Update check complete"
|
SSH Authorized Keys Audit
Detect unauthorized SSH keys — run weekly:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
|
#!/usr/bin/env bash
set -euo pipefail
# ssh-key-audit.sh — Audit all authorized_keys files on the system
REPORT_FILE="/var/log/ssh-key-audit-$(date +%Y%m%d).txt"
{
echo "=== SSH Authorized Keys Audit — $(date) ==="
echo "Host: $(hostname -f)"
echo ""
# Check root's authorized keys
if [[ -f /root/.ssh/authorized_keys ]]; then
echo "--- root ---"
while IFS= read -r key; do
[[ -z "$key" || "$key" =~ ^# ]] && continue
key_type=$(echo "$key" | awk '{print $1}')
key_comment=$(echo "$key" | awk '{print $3}')
fingerprint=$(echo "$key" | ssh-keygen -lf /dev/stdin 2>/dev/null | awk '{print $2, $3, $4}')
echo " [$key_type] $key_comment — $fingerprint"
done < /root/.ssh/authorized_keys
echo ""
fi
# Check all users with UID >= 1000
while IFS=: read -r user _ uid _ _ home _; do
[[ "$uid" -lt 1000 ]] && continue
auth_keys="${home}/.ssh/authorized_keys"
[[ -f "$auth_keys" ]] || continue
echo "--- $user (uid $uid) ---"
while IFS= read -r key; do
[[ -z "$key" || "$key" =~ ^# ]] && continue
key_type=$(echo "$key" | awk '{print $1}')
key_comment=$(echo "$key" | awk '{print $3}')
fingerprint=$(echo "$key" | ssh-keygen -lf /dev/stdin 2>/dev/null | awk '{print $2, $3, $4}')
echo " [$key_type] $key_comment — $fingerprint"
done < "$auth_keys"
# Warn on insecure permissions
perms=$(stat -c "%a" "$auth_keys")
if [[ "$perms" != "600" && "$perms" != "644" ]]; then
echo " WARNING: Insecure permissions on $auth_keys: $perms (should be 600)"
fi
echo ""
done < /etc/passwd
echo "=== End of Audit ==="
} | tee "$REPORT_FILE"
echo "Report saved to: $REPORT_FILE"
|
Cron Job Inventory
Document every scheduled task on the system — invaluable for audits:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
#!/usr/bin/env bash
set -euo pipefail
# cron-inventory.sh — List all cron jobs across all users and system files
echo "=== Cron Job Inventory — $(hostname) — $(date '+%Y-%m-%d') ==="
echo ""
# System crontab
echo "--- /etc/crontab ---"
grep -v '^#\|^$' /etc/crontab 2>/dev/null || echo " (empty)"
echo ""
# /etc/cron.d/
echo "--- /etc/cron.d/ ---"
for f in /etc/cron.d/*; do
[[ -f "$f" ]] || continue
echo " File: $f"
grep -v '^#\|^$' "$f" | while read -r line; do echo " $line"; done
done
echo ""
# System cron directories
for dir in /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly; do
if [[ -d "$dir" ]] && [[ -n "$(ls -A "$dir")" ]]; then
echo "--- $dir ---"
ls -1 "$dir"
echo ""
fi
done
# Per-user crontabs
echo "--- User Crontabs ---"
for user in $(cut -d: -f1 /etc/passwd); do
crontab_content=$(crontab -l -u "$user" 2>/dev/null | grep -v '^#\|^$' || true)
if [[ -n "$crontab_content" ]]; then
echo " User: $user"
echo "$crontab_content" | while read -r line; do echo " $line"; done
echo ""
fi
done
# Systemd timers
echo "--- Systemd Timers ---"
systemctl list-timers --all --no-pager 2>/dev/null \
| grep -v '^$' \
| head -30
|
Network and Connectivity Scripts
Port Scanner / Open Port Audit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
#!/usr/bin/env bash
set -euo pipefail
# open-ports.sh — Show all listening ports with owning process
echo "=== Listening Ports on $(hostname) — $(date '+%Y-%m-%d %H:%M') ==="
echo ""
printf "%-10s %-8s %-25s %-10s %s\n" "PROTO" "PORT" "ADDRESS" "PID" "PROCESS"
printf "%-10s %-8s %-25s %-10s %s\n" "-----" "----" "-------" "---" "-------"
ss -tlnup 2>/dev/null | tail -n +2 | while read -r state recvq sendq local_addr peer_addr process; do
proto="TCP"
port=$(echo "$local_addr" | rev | cut -d: -f1 | rev)
addr=$(echo "$local_addr" | rev | cut -d: -f2- | rev)
pid=$(echo "$process" | grep -oP 'pid=\K[0-9]+' || echo "-")
name=$(echo "$process" | grep -oP 'users:\(\("\K[^"]+' || echo "unknown")
printf "%-10s %-8s %-25s %-10s %s\n" "$proto" "$port" "$addr" "$pid" "$name"
done
ss -ulnup 2>/dev/null | tail -n +2 | while read -r state recvq sendq local_addr peer_addr process; do
proto="UDP"
port=$(echo "$local_addr" | rev | cut -d: -f1 | rev)
addr=$(echo "$local_addr" | rev | cut -d: -f2- | rev)
pid=$(echo "$process" | grep -oP 'pid=\K[0-9]+' || echo "-")
name=$(echo "$process" | grep -oP 'users:\(\("\K[^"]+' || echo "unknown")
printf "%-10s %-8s %-25s %-10s %s\n" "$proto" "$port" "$addr" "$pid" "$name"
done
|
Network Connectivity Checker
Useful as a boot-time or cron check on isolated servers:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
#!/usr/bin/env bash
# connectivity-check.sh — Verify network connectivity to critical hosts
declare -A TARGETS=(
["Gateway"]="10.0.0.1"
["DNS"]="8.8.8.8"
["Internal DB"]="10.0.1.100"
["External"]="1.1.1.1"
)
DNS_TEST="example.com"
all_ok=true
echo "=== Network Connectivity Check — $(date '+%H:%M:%S') ==="
for name in "${!TARGETS[@]}"; do
ip="${TARGETS[$name]}"
if ping -c1 -W2 "$ip" &>/dev/null; then
printf " %-20s %-15s %s\n" "$name" "$ip" "OK"
else
printf " %-20s %-15s %s\n" "$name" "$ip" "FAIL"
all_ok=false
fi
done
# DNS resolution check
echo ""
if host "$DNS_TEST" &>/dev/null; then
echo " DNS resolution: OK ($DNS_TEST resolves)"
else
echo " DNS resolution: FAIL (cannot resolve $DNS_TEST)"
all_ok=false
fi
"$all_ok" && echo "" && echo "All checks passed." || echo "" && echo "Some checks FAILED."
|
System Snapshot and Baseline
Daily System Snapshot
Run from cron to capture system state for forensics and drift detection:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
#!/usr/bin/env bash
set -euo pipefail
# system-snapshot.sh — Capture system state for baselining/forensics
SNAPSHOT_DIR="/var/log/snapshots/$(date '+%Y%m%d_%H%M%S')"
mkdir -p "$SNAPSHOT_DIR"
capture() {
local name="$1"; shift
"$@" > "${SNAPSHOT_DIR}/${name}.txt" 2>&1 || true
}
echo "Capturing system snapshot to $SNAPSHOT_DIR..."
capture "01_uname" uname -a
capture "02_uptime" uptime
capture "03_users" who
capture "04_processes" ps auxf
capture "05_netstat" ss -tlnup
capture "06_disk_usage" df -h
capture "07_disk_inodes" df -i
capture "08_mounts" cat /proc/mounts
capture "09_memory" free -h
capture "10_top_memory" ps aux --sort=-%mem
capture "11_top_cpu" ps aux --sort=-%cpu
capture "12_open_files" lsof -n 2>/dev/null
capture "13_kernel_log" dmesg -T
capture "14_failed_units" systemctl --failed
capture "15_last_logins" last -50
capture "16_auth_failures" grep "Failed password" /var/log/auth.log 2>/dev/null \
|| journalctl -u sshd --since "24 hours ago" --no-pager
capture "17_crontabs" crontab -l 2>/dev/null
capture "18_listening" ss -tlnup
capture "19_arp_cache" arp -n
capture "20_routes" ip route show
# Checksums of critical system files
find /etc /usr/bin /usr/sbin -type f 2>/dev/null \
| xargs md5sum 2>/dev/null \
> "${SNAPSHOT_DIR}/checksums.txt"
echo "Snapshot complete: $SNAPSHOT_DIR"
echo "Files captured: $(ls "$SNAPSHOT_DIR" | wc -l)"
# Keep only last 30 days of snapshots
find "$(dirname "$SNAPSHOT_DIR")" -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \; 2>/dev/null || true
|
Putting It All Together: A Weekly Sysadmin Report
Combine the above into a single weekly digest emailed to your team:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
|
#!/usr/bin/env bash
set -euo pipefail
# weekly-report.sh — Comprehensive weekly system health report
REPORT_DATE="$(date '+%Y-%m-%d')"
HOSTNAME="$(hostname -f)"
ALERT_EMAIL="ops@example.com"
WEEK_AGO="7 days ago"
{
echo "Subject: Weekly System Report — $HOSTNAME — $REPORT_DATE"
echo ""
echo "========================================"
echo " Weekly System Report"
echo " Host: $HOSTNAME"
echo " Generated: $(date)"
echo "========================================"
echo ""
echo "--- UPTIME & LOAD ---"
uptime
echo ""
echo "--- DISK USAGE ---"
df -h | grep -v tmpfs
echo ""
echo "--- MEMORY ---"
free -h
echo ""
echo "--- FAILED SERVICES ---"
systemctl --failed --no-pager 2>/dev/null || echo " (none)"
echo ""
echo "--- TOP 10 MEMORY CONSUMERS ---"
ps aux --sort=-%mem --no-headers | head -10 \
| awk '{printf " %-20s %5s%% RSS PID=%s\n", $11, $4, $2}'
echo ""
echo "--- USER LOGINS (last 7 days) ---"
last -s "$WEEK_AGO" | grep -v "^reboot\|^wtmp\|^$" | head -20
echo ""
echo "--- SSH FAILURES (top IPs) ---"
journalctl -u sshd --since "$WEEK_AGO" --no-pager 2>/dev/null \
| grep "Failed password" \
| grep -oP 'from \K[0-9.]+' \
| sort | uniq -c | sort -rn | head -10 \
| awk '{printf " %6d attempts from %s\n", $1, $2}' \
|| echo " (no data)"
echo ""
echo "--- PACKAGE UPDATES AVAILABLE ---"
if command -v apt-get &>/dev/null; then
apt-get -s upgrade 2>/dev/null | grep "^[0-9]" | head -3
elif command -v dnf &>/dev/null; then
dnf check-update -q 2>/dev/null | wc -l | xargs -I{} echo " {} packages available"
fi
echo ""
echo "--- RECENT KERNEL ERRORS ---"
journalctl -k --since "$WEEK_AGO" -p err --no-pager 2>/dev/null \
| grep -v '^-- ' | tail -20 \
|| echo " (none)"
echo ""
echo "========================================"
echo " End of Report"
echo "========================================"
} | sendmail "$ALERT_EMAIL"
echo "Weekly report sent to $ALERT_EMAIL"
|
Add to root’s crontab for Monday morning delivery:
0 7 * * 1 /usr/local/bin/weekly-report.sh
Script Deployment Tips
Installing Scripts System-Wide
1
2
3
4
5
6
7
|
# Copy scripts to a standard location
cp disk-alert.sh /usr/local/bin/disk-alert
chmod 755 /usr/local/bin/disk-alert
chown root:root /usr/local/bin/disk-alert
# For root-only scripts
chmod 700 /usr/local/bin/offboard-user
|
Cron Scheduling Reference
# ┌─ minute (0-59)
# │ ┌─ hour (0-23)
# │ │ ┌─ day of month (1-31)
# │ │ │ ┌─ month (1-12)
# │ │ │ │ ┌─ day of week (0=Sun, 6=Sat)
# │ │ │ │ │
* * * * * command
*/15 * * * * Every 15 minutes
0 * * * * Every hour
0 2 * * * Daily at 2:00 AM
0 2 * * 0 Weekly on Sunday at 2:00 AM
0 2 1 * * First of each month at 2:00 AM
0 7 * * 1 Every Monday at 7:00 AM
Using systemd Timers Instead of Cron
For modern systems, systemd timers are more reliable — they log to the journal, handle missed runs, and show status with systemctl:
1
2
3
4
5
6
7
|
# /etc/systemd/system/disk-alert.service
[Unit]
Description=Disk Space Alert Check
[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-alert
|
1
2
3
4
5
6
7
8
9
10
|
# /etc/systemd/system/disk-alert.timer
[Unit]
Description=Run disk-alert every 15 minutes
[Timer]
OnCalendar=*:0/15
Persistent=true
[Install]
WantedBy=timers.target
|
1
2
3
|
systemctl enable --now disk-alert.timer
systemctl list-timers disk-alert.timer
journalctl -u disk-alert.service
|
A well-maintained library of sysadmin scripts is one of the most valuable assets a team can build. Start with the tasks you do manually every week, automate them one at a time, and invest in the logging and alerting that turns a silent cron job into a visible, auditable system process.
Comments