The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):
"""
The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.
If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.
Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.
sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)
Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.
[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.
[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...
journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.
It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:
[Service]
LogFilterPatterns=~I am a completely useless log entry
LogFilterPatterns=~I am another useless log entry
But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
- workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717
- kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030
- kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805
- bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192
- firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe
- ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316
- mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405
> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes
Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
Yeah docker does a ton of network stuff when you start/stop containers, depending on your configuration. It's extra fun because it can drop existing connections when that happens.
Had a process quietly in a crash loop for a solid month on my workstation until I figured out what was causing my random network outages.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
Thank you for your work. ISTM the workload is naturally LSM-shaped.
> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.
> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).
If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.
Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.
If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.
In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.
What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.
I’ve met people that think the log should be remotely stored and not written locally, since it’ll be shipped to splunk or whatever anyway.
Those people change their minds the first time a machine has intermittent network issues, and the logs needed to debug it are lost (or worse, the log buffer fills, then stdout fills, which backpressures the application, creating an outage while simultaneously eating the logs).
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
What is the real-world use for these features? Who is this built for?
Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.
I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.
Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.
Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
> syslog doesn't have nearly the functionality that journald + journalctl does.
A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.
Also, in regards to your original comment:
> This issue report feels like it ought to be accompanied by a fix.
This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!
So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.
[0] AIUI, it came out of Google, but my understanding might be incorrect.
[1] ...regardless of whether or not that effect is intentional...
Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.
(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)
Systemd things being horse shit as usual because it was vibecoded even before LLM existed.
And there are still people that said that systemd and tools are awesome because they never encountered any of the countless ridicule bugs.
Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd should just use DuckDB. It's perfect for this job.
"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"
Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.
BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.
Seems like there are a lot of mistaken impressions about DB storage engines out there.
Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.
You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
reply