Repository navigation
Bugfix + features for lessening forwarding load - #153
Merged
Merged
Conversation
Looks like the refactor in 57b0864 caused all messages to be discarded by making parse_journal_fields return 0 on success, but then also skipping the message when it does in journal_read_input. This looks to get it working again.
Add a new config option to exclude systemd units from log forwarding. Note this is intended for a small number of units, not hundreds, so it's just a linearly compared list not anything more fancy.
Add two new options to pause when we've just sent X lines in a tight loop (a "chunk"). This allows the cursor to be updated on disk, send the systemd watchdog notification/ping, and avoid overloading the receiver on first start (where we may be trying to transmit millions of historical log lines). Better show the status in systemctl status. Previously would often just show "idle". Now should say if its ready, idle (disconnected), or paused (inter-chunk delay). Documentation included.
Sorry about that, I missed one spot when I changed the 10s I was testing with to 1s. Was just the example config, but that'd be confusing, so fixing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two features to help reduce load on the forwarding destinations:
First commit is a fix for #149.
Changes
ExcludeUnits, which is a space-separated list of systemd unit names to exclude from forwarding. Note I implemented this expecting a short list — presumably just the one or two that are producing a lot of noise. That keeps the code simple, but performance will suffer if someone tries to list hundreds. Default to not exclude any.ChunkMaxLineCount, default 100. This controls the maximum number of journal lines that'll be handled in the tight read-send loop. After that many lines, it'll save the state file, and use sd_event_add_time to pause for a bit (configurable byChunkDelaySec)ChunkDelaySec, the delay for aboveNote that the previous behavior without the chunking was quite bad if you installed netlogd with a substantial existing journal (which you want to send). It'd try to send the whole thing, which could be many millions of lines, in that loop. Since it wasn't returning to the main event loop, the systemd watchdog didn't get notified — so systemd would eventually kill it. And also since it wasn't returning, the state file wasn't updated with the new cursor position. So if you restarted it, it'd start over from the beginning. The chunking fixes that.
The default are just numbers I picked; I guessed 100 lines + one second is probably high enough that it won't happen, except during the initial send after installation. And if it does happen, very few installs would be unable to keep up because of the delay.
It might be possible to lower the watchdog timeout with this.
Testing
meson test -C buildpassesChecklist