Is this how logs are supposed to look? -- Boot ca9164cc44384ec8aea2470b59e9681b --

[webmogul1]

This is part of Proxmox System log.

Jun 26 11:23:43 pve01 kernel: vmbr2: port 8(veth180i0) entered disabled state
-- Boot ca9164cc44384ec8aea2470b59e9681b --
Jun 26 11:41:12 pve01 systemd[1]: Started session-13.scope - Session 13 of User root.
-- Boot 0fc3e6dca6aa4074a2a9def664299fb1 --
Jun 26 11:23:43 pve01 kernel: veth180i0 (unregistering): left allmulticast mode
-- Boot ca9164cc44384ec8aea2470b59e9681b --
Jun 26 11:41:12 pve01 login[21332]: ROOT LOGIN  on '/dev/pts/0'
-- Boot 0fc3e6dca6aa4074a2a9def664299fb1 --
Jun 26 11:23:43 pve01 kernel: veth180i0 (unregistering): left promiscuous mode
Jun 26 11:23:43 pve01 kernel: vmbr2: port 8(veth180i0) entered disabled state
Jun 26 11:23:43 pve01 kernel: vmbr2: port 3(veth110i0) entered disabled state
Jun 26 11:23:43 pve01 kernel: veth110i0 (unregistering): left allmulticast mode
Jun 26 11:23:43 pve01 kernel: veth110i0 (unregistering): left promiscuous mode
Jun 26 11:23:43 pve01 kernel: vmbr2: port 3(veth110i0) entered disabled state
Jun 26 11:23:43 pve01 systemd[1]: pve-container@160.service: Deactivated successfully.

[azlux]

What is this ticket ? It's not related to log2ram.

[doloopuntil]

I think @webmogul1 is onto something. After having installed log2ram on my Debian Trixie, I am getting a jumbled up system log

...
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: proc-sys-fs-binfmt_misc.mount: Deactivated successfully.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:     'statsd.bad_lines_seen': 0,
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Unset automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   counter_rates: {
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: proc-sys-fs-binfmt_misc.automount: Deactivated successfully.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   timer_data: {},
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Stopped systemd-binfmt.service - Set Up Additional Binary Formats.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   gauges: { 'statsd.timestamp_lag': 0 },
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: systemd-binfmt.service: Deactivated successfully.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   timers: {},
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Stopped systemd-timesyncd.service - Network Time Synchronization.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   },
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: systemd-timesyncd.service: Deactivated successfully.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:     'statsd.metrics_received': 0
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd-networkd[509]: enp7s0: DHCPv6 lease lost
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:     'statsd.packets_received': 0,
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd-networkd[509]: enp7s0: DHCP lease lost
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:     'statsd.bad_lines_seen': 0,
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Stopped systemd-tmpfiles-setup.service - Create System Files and Directories.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]:   counters: {
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully.
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]: {
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Stopping systemd-timesyncd.service - Network Time Synchronization...
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:14:53 fresh-footprints-main sh[1116]: Flushing stats at  Mon Aug 25 2025 05:14:53 GMT+0000 (Coordinated Universal Time)
-- Boot 0b9557ec5fed4c169b75fb363c1c4408 --
Aug 25 20:46:48 fresh-footprints-main systemd[1]: Stopping systemd-random-seed.service - Load/Save OS Random Seed...
-- Boot 22d2b5e791ea47efbad94344e0e24e24 --
Aug 25 05:05:01 fresh-footprints-main CRON[16104]: pam_unix(cron:session): session closed for user root
...

where log entries for different system boots (boots at different times) seem to be interleaved, instead of being grouped together.

[azlux]

indeed, it's look like the journactl havn't closed the previous boot logs. So the database mess up the logs.
Can you run a journalctl --verify if any errors ?

[JVKeller]

@azlux If it helps... I noticed this as well.

Can you run a journalctl --verify if any errors ? ALL PASS.

Upgraded Proxmox from 8 -> 9, and noticed the log2ram service was no longer loading. After seeing the log was just a little bit on the larger size 🙄 I vacuumed it and made some minor adjustments and right after the server was restarted, the entries started to get scrambled.

See right after log2ram finished scanning the files, it started jumbling.

Sep 09 15:20:07 pve systemd[1]: Starting log2ram.service - Log2Ram...
Sep 09 15:20:07 pve log2ram[26327]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Sep 09 15:20:07 pve log2ram[26327]: File(s) causing issues
Sep 09 15:20:07 pve log2ram[26327]: : 1.6G        /var/hdd.log/journal
Sep 09 15:20:07 pve systemd[1]: var-log.mount: Deactivated successfully.
Sep 09 15:20:07 pve log2ram[26355]: /usr/bin/log2ram: line 72: mail -s Log2Ram Error on pve root: command not found
Sep 09 15:20:07 pve systemd[1]: var-hdd.log.mount: Deactivated successfully.
Sep 09 15:20:07 pve systemd[1]: log2ram.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 15:20:07 pve systemd[1]: log2ram.service: Failed with result 'exit-code'.
Sep 09 15:20:07 pve systemd[1]: Failed to start log2ram.service - Log2Ram.
Sep 09 15:24:38 pve systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Sep 09 15:24:38 pve systemd-tmpfiles[30949]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
Sep 09 15:24:38 pve systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Sep 09 15:24:38 pve systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Sep 09 15:28:51 pve systemd[1]: Starting log2ram.service - Log2Ram...
Sep 09 15:28:51 pve log2ram[35242]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Sep 09 15:28:51 pve log2ram[35242]: File(s) causing issues
Sep 09 15:28:51 pve log2ram[35242]: : 500M        /var/hdd.log/journal
Sep 09 15:28:51 pve systemd[1]: var-log.mount: Deactivated successfully.
Sep 09 15:28:51 pve log2ram[35261]: /usr/bin/log2ram: line 72: mail -s Log2Ram Error on pve root: command not found
Sep 09 15:28:51 pve systemd[1]: var-hdd.log.mount: Deactivated successfully.
Sep 09 15:28:51 pve systemd[1]: log2ram.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 15:28:51 pve systemd[1]: log2ram.service: Failed with result 'exit-code'.
Sep 09 15:28:51 pve systemd[1]: Failed to start log2ram.service - Log2Ram.
Sep 09 15:29:23 pve systemd[1]: Starting log2ram.service - Log2Ram...
Sep 09 15:29:23 pve log2ram[35869]: building file list ... done
Sep 09 15:29:23 pve log2ram[35869]: README -> ../../usr/share/doc/systemd/README.logs

     ...scanned all the log files...

Sep 10 23:55:53 pve log2ram[600547]: log2ram.log
Sep 10 23:55:53 pve log2ram[600547]: mail.log
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:06:03 pve systemd-journald[1164]: System Journal (/var/log/journal/f2e0eb27effe404894a02dec1bbeae1d) is 41.6M, max 211.8M, 170.2M fr>
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: syslog
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:06:03 pve systemd-journald[1164]: Received client request to rotate journal, rotating.
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: user.log
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:06:03 pve systemd-journald[1164]: Vacuuming done, freed 0B of archived journals from /var/log/journal/f2e0eb27effe404894a02dec1bbeae>
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: wtmp
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:46 pve pvedaemon[1756]: <root@pam> starting task UPID:pve:00008A05:0002615C:68C31E02:spiceshell::root@pam:
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: ifupdown2/
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:46 pve pvedaemon[35333]: starting spiceterm UPID:pve:00008A05:0002615C:68C31E02:spiceshell::root@pam: - Shell on 'pve'
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: ifupdown2/network_config_ifupdown2_215_Sep-10-2025_07:10:27.655292/
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:46 pve pvedaemon[35333]: launch command: /usr/bin/spiceterm --port 61000 --addr localhost --timeout 40 --authpath /nodes/pve --pe>
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: ifupdown2/network_config_ifupdown2_215_Sep-10-2025_07:10:27.655292/ifupdown2.debug.log
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:46 pve login[35337]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: ifupdown2/network_config_ifupdown2_215_Sep-10-2025_07:10:27.655292/interfaces
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:47 pve systemd-logind[1290]: New session 9 of user root.
-- Boot 3fc9ad9a1b874707a11b12a9586600cc --
Sep 10 23:55:53 pve log2ram[600547]: ifupdown2/network_config_ifupdown2_215_Sep-10-2025_07:10:27.655292/interfaces.d/
-- Boot f74e5cf9c1f6444bab176f9efef39af2 --
Sep 11 15:07:47 pve systemd[1]: Started session-9.scope - Session 9 of User root.

[JVKeller]

Update...

It seems to be ok after some log rotations and double checking size settings and restating the service.

My last few lines for reference only:

Sep 12 08:16:39 pve root[1047202]: Testing log2ram and journald
Sep 12 08:17:01 pve CRON[1047560]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Sep 12 08:17:01 pve CRON[1047562]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 12 08:17:01 pve CRON[1047560]: pam_unix(cron:session): session closed for user root
Sep 12 08:27:15 pve pveproxy[1023258]: worker exit
Sep 12 08:27:15 pve pveproxy[1764]: worker 1023258 finished
Sep 12 08:27:15 pve pveproxy[1764]: starting 1 worker(s)
Sep 12 08:27:15 pve pveproxy[1764]: worker 1058886 started
Sep 12 08:29:48 pve pveproxy[1022648]: worker exit
Sep 12 08:29:48 pve pveproxy[1764]: worker 1022648 finished
Sep 12 08:29:48 pve pveproxy[1764]: starting 1 worker(s)
Sep 12 08:29:48 pve pveproxy[1764]: worker 1061746 started
Sep 12 08:30:41 pve root[1062831]: Testing log2ram and journald
Sep 12 08:30:55 pve pvedaemon[1031158]: <root@pam> successful auth for user 'root@pam'
Sep 12 08:32:29 pve pvestatd[1728]: auth key pair too old, rotating..
Sep 12 08:38:18 pve systemd[1]: Stopping pveproxy.service - PVE API Proxy Server...
Sep 12 08:38:19 pve pveproxy[1764]: received signal TERM
Sep 12 08:38:19 pve pveproxy[1764]: server closing

[azlux]

Sorry,
I will check on it next week (holidays right now)

I suppose journactl havn't finish writing logs when log2ram stop the mount point at restart.
I will check if journald have a command to force sync/write.

[azlux]

Is that possible to test with the command :
journalctl --sync
before the cp/rsync command here : https://github.com/azlux/log2ram/blob/master/log2ram#L76

[JVKeller]

@azlux I had ran that command at one point to try and get things re synced. Whether that is what corrected it for me, I will never know. :)

Maybe @webmogul1 can test that if he's still having issues.

[webmogul1]

@JVKeller I am sorry, I ended up not using log2ram.

[JVKeller]

Just wanted to mention that the ordering is still not correct.

[XhmikosR]

To whoever is hitting this, can you set SystemMaxUse? I had similar issues and this seems to do the job.