log2ram and journal

[Sailing74]

Could it be that, despite this previous issue/fix, there is still an issue with journal and ACLs, probably because of some sort of racing condition upon system boot?

From time to time, I am encountering situations where "journalctl" shows the log from BEFORE the last boot, only, while "journalctl -b" shows the log from AFTER the last boot - suggesting that there is an issue with writing the current journal onto log2ram. Usually, this is fixed by rebooting.

I just have a vm where this happened again - and I am having some time to report it. I compared /var/log/journal on both systems - permissions and acls are identical, but journald appears to be still not able to write the journal on one VM.

# systemd-analyze critical-chain systemd-journald.service

systemd-journald.service +50ms
└─log2ram.service @1.563s +360ms
  └─var-log.mount @2.531s
    └─local-fs-pre.target @631ms
      └─systemd-tmpfiles-setup-dev.service @610ms +20ms
        └─systemd-tmpfiles-setup-dev-early.service @582ms +19ms
          └─kmod-static-nodes.service @453ms +78ms
            └─systemd-journald.socket @444ms
              └─system.slice @395ms
                └─-.slice @395ms

If I read this correctly, it means that systemd-journald.socket actually started before log2ram, which could explain the difficulties in writing. And this is even though the log2ram.service file contains:

Before=basic.target rsyslog.service syslog-ng.service syslog.target systemd-journald.service sysinit.target shutdown.target zram-swap-conf.service apache2.service lighttpd.service

Assuming this means that there is indeed a race condition, would it help to add

[Unit]
After=log2ram.service

to systemd-journald.service as well as part of installing log2ram, e.g. by creating /etc/systemd/system/systemd-journald.service.d/log2ram.conf with this content? Or will it break other things?

[Sailing74]

In case it should help others: I was able to solve the issue by adding the following to log2ram's configuration as /etc/systemd/system/log2ram.service.d/ordering.conf:

[Unit]
DefaultDependencies=no
Before=systemd-journald.service systemd-journald.socket sysinit.target
Wants=var-log.mount
After=local-fs-pre.target

[Sailing74]

Ok, the above approach still leads to race conditions here.

I appear to be running into issues on Debian 13 in the sense that I fail to enable two different tasks within log2ram's systemd service: (1) Mount /var/log as tmpfs (2) Sync logs from RAM back to disk reliably before journald starts writing.

As far as I understand, log2ram mounting /var/log inside a service unit races against systemd-journald (which starts very early via sockets), systemd-remount-fs.service (root still read-only), systemd-tmpfiles-setup.service, and sysinit.target ordering.

Typically, mounting belongs to the filesystem boot phase (handled by systemd mount units) and syncing belongs to the service phase. Issues I've now experience: journald writing to ext4 before log2ram mounts tmpfs, log2ram starting too early while / is still read-only and then journald failing to start, systemd ordering cycles and broken logging and unstable boots.

It seems to me that the best way forward might be to split responsibilities: (1) A dedicated /var/log.mount unit mounts tmpfs deterministically before journald. (2) A log2ram-sync.service unit performing log2ram write or sync actions.

Would this be a better setup?

Meanwhile, I've pulled together the below log2ram.service, which limits race conditions here for me on Debian 13, but doesn't always reliably avoid them.

[Unit]
Description=Log2Ram (fixed ordering)
Documentation=https://github.com/azlux/log2ram
DefaultDependencies=yes
After=local-fs.target systemd-remount-fs.service systemd-tmpfiles-setup.service
RequiresMountsFor=/var/log

[Service]
Type=oneshot
ExecStart=/usr/bin/log2ram start
ExecStop=/usr/bin/log2ram stop
ExecReload=/usr/bin/log2ram write
TimeoutStartSec=120
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target