Think parsing logs in Fluentd is harder than it’s worth?
It’s not. Pick the right parser and settings and parsing gets easy.
This guide gives the exact settings, short examples, and real test steps to parse common logs (regexp, json, apache/nginx, syslog) and handle multiline stack traces.
You’ll get the key fields to set – keyname, reservedata, @type, expression, timekey and timeformat – plus troubleshooting tips so your parser behaves in production.
Read on to apply a working Fluentd log parser configuration in minutes.
Core Fundamentals of the Fluentd Log Parser for Immediate Implementation
Parsing in Fluentd turns messy log messages into clean, structured records with actual field names. You get a raw text line like “2024-01-15 14:32:10 ERROR [Auth] Login failed,” and parsing pulls out each piece—timestamp, level, module, message—into separate fields you can index, route, and search. Apply parser config in a <filter> block when you’re processing events mid-flight, or drop it in a <source> block to parse right at ingestion.
Your key parser options control how Fluentd grabs fields. Set key_name to whatever record field you want to parse. Usually that’s key_name message because you’re targeting the raw message body. Add reserve_data true if you want to keep the original message sitting alongside your extracted fields for debugging or audit trails. Inside the <parse> block, you’ll specify @type regexp and hand it an expression with Ruby-style named capture groups like (?<level>[A-Z]+) or (?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}). Each named group becomes a field in your output record automatically.
Timestamp extraction needs the time_key parameter pointing at your captured timestamp field, plus time_format matching your pattern exactly. So time_format %Y-%m-%d %H:%M:%S matches “2024-01-15 14:32:10.” When logs don’t match, Fluentd throws a warning like pattern not matched: "Hello World" in the agent log. Restart Fluentd or td-agent after every config change to actually apply your new parsing rules.
Required parser settings if you’re new to Fluentd:
key_name message— parse the message fieldreserve_data true— keep the original raw log line around@type regexp— use regular expression parserexpression /(?<field>pattern)/— define your named capture groupstime_keyandtime_format— pull out and parse timestamps correctly
Configuring Fluentd Log Parser Plugins for Regex, JSON, Apache, Nginx, and Syslog Formats
Which parser plugin you pick depends on your log source and how consistent the format is. Built-in parsers for json, apache2, nginx, syslog, csv, tsv, and ltsv handle standard structures. The regexp parser gives you full control for custom patterns.
Regex Parser
Use the regexp parser when your logs follow some consistent custom format that doesn’t fit the built-in parsers. You’ll define named capture groups like (?<log_source>[^']+) to grab fields by position or delimiter. Regex parsing is flexible but it’s CPU-intensive, so keep your expressions specific and avoid backtracking. You need this for proprietary application logs or whatever bespoke middleware output you’re stuck with.
JSON Parser
Structured JSON logs are the fastest thing you can parse because Fluentd just deserializes the payload without any pattern matching. Use @type json when your applications spit out JSON natively. Something like {"level":"ERROR","module":"Auth","message":"Login failed"} parses instantly with zero regex overhead. For high-throughput pipelines, prefer structured logging.
Web Server Parsers (Apache/Nginx)
Apache and Nginx parsers come with predefined field mappings for common access log formats. The apache2 parser knows combined log format with remote_addr, request, status, bytes. Nginx parser handles the default nginx access log structure. Use these when you control the web server log format and you want zero-config parsing.
Syslog Parsers
Fluentd’s syslog parser supports rfc3164 (older BSD syslog), rfc5424 (newer structured), or message_format auto to accept both. Pick rfc5424 for modern network devices and systemd. Use rfc3164 for legacy systems. Set with_priority false if your incoming messages don’t have the priority field, or switch to the tcp input plugin for non-standard syslog streams.
| Parser | Ideal Use Case | Notes |
|---|---|---|
| regexp | Custom application logs | Ruby-style named captures; CPU cost |
| json | Structured logging | Best performance; zero pattern overhead |
| apache2/nginx | Standard web server access logs | Predefined fields; no config needed |
| syslog | Network devices, OS logs | Supports rfc3164, rfc5424, auto |
Fluentd Log Parser Configuration Examples for Common Log Types
Named capture groups in your regular expression map straight to record fields. Fluentd runs the expression against whatever’s in key_name, pulls out each (?<fieldname>pattern), and builds a new record with those keys. Input like log_source='app1' index='prod' paired with (?<log_source>[^']+) and (?<index>[^']+) gives you output with log_source: "app1" and index: "prod".
Log Source / Index Extraction
Pull metadata fields from log preambles. Given input log_source='frontend' index='us-west', use expression /log_source='(?<log_source>[^']+)' index='(?<index>[^']+)'/ to capture both. The negated character class [^']+ matches everything up to the closing single quote. You get two clean fields for routing and filtering.
Timestamp / Level / Module Extraction
Structured application logs usually start with timestamp, severity level, and component. For input “2024-01-15 14:32:10 ERROR [Auth] Invalid token,” use:
expression /(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<level>[A-Z]+) \[(?<module>[^\]]+)\] (?<message>.*)/
time_key timestamp
time_format %Y-%m-%d %H:%M:%S
The \d{4}-\d{2}-\d{2} matches year-month-day, [A-Z]+ grabs all-caps levels like ERROR or INFO, [^\]]+ gets text inside brackets, and .* takes whatever’s left as message. time_key tells Fluentd which field holds the event timestamp, time_format defines how to parse that string.
Key‑Value Pair Extraction
Application logs with space-separated key-value pairs like “user=alice status=success ip=192.168.1.5” parse with (?<user>\w+), (?<status>\w+), (?<ip>[^\s]+). Full expression:
/user=(?<user>\w+) status=(?<status>\w+) ip=(?<ip>[^\s]+)/
Each \w+ captures word characters and [^\s]+ grabs non-whitespace for the IP. Your output record gets user, status, and ip as separate searchable fields.
Steps to test each configuration:
- Add the parser filter to
/etc/td-agent/td-agent.confwith the correct@typeandexpression. - Restart td-agent:
sudo systemctl restart td-agent. - Send sample logs to the configured input (file tail, tcp port, or forward).
- Check
/var/log/td-agent/td-agent.logfor parse errors or “pattern not matched” warnings.
Multiline Fluentd Log Parser Strategies for Stack Traces and Multi‑Row Logs
Application stack traces and exception logs span multiple lines but they’re really a single event. Parsing each line individually breaks context. Java exceptions, Python tracebacks, multi-row JSON payloads need multiline handling to group continuation lines with the initial message.
Fluentd’s multiline parser handles limited line-continuation scenarios. For complex multi-event streams, look at fluent-plugin-concat or fluent-plugin-multi-format-parser. Basic multiline works when the first line of each event follows a recognizable pattern like a timestamp or severity marker.
Configuring format_firstline and Continuation Patterns
Use format_firstline to spot the start of a new record. For logs beginning with a timestamp, set format_firstline /^\d{4}-\d{2}-\d{2}/ to match lines starting with a date. Then use format1 to capture the entire message including continuation lines: format1 /(?<message>.*)/. Fluentd buffers lines until it sees another match for format_firstline, then emits the buffered block as a single message field. This groups a stack trace starting with “2024-01-15 ERROR Exception occurred” and continuing with indented “at com.example…” lines into one record.
Validation steps for multiline parsing:
- Confirm the
format_firstlineregex matches only the first line of each event. - Test with sample multi-row input to verify continuation lines get appended, not dropped.
- Check td-agent logs for “pattern not matched” if some lines are getting rejected.
- Use
@type stdoutin a match block to inspect the finalmessagefield and confirm all lines are captured.
Troubleshooting Fluentd Log Parser Errors and Invalid Record Handling
Parsing errors happen when the log line structure doesn’t match your configured regular expression. Misaligned delimiters, unexpected spacing, missing fields trigger a mismatch. Fluentd spits out a warning record tagged fluent.warn with the original message content, telling you the parser couldn’t extract fields.
Fluentd warning messages look like 2023-07-21 00:14:57 +0000 fluent.warn: {"message":"pattern not matched: \"Hello World\""} and they show up in the agent log or as routed events if you’re forwarding fluent.** tags. Each unmatched log creates a new warning, cluttering your output. Check that your capture groups account for all expected delimiters and that spacing in the regex matches input exactly. A single extra space breaks the match.
To suppress warnings for records that don’t parse, add emit_invalid_record_to_error false in your parser configuration. This only works inside a <filter> directive, not in <source>. If you set it in a source block, Fluentd ignores it and warnings keep coming. Place the parser filter after the input and include emit_invalid_record_to_error false to silently drop unmatched records instead of emitting fluent.warn events.
| Issue | Cause | Fix |
|---|---|---|
| Pattern not matched | Regex doesn’t align with log structure | Test regex with sample logs; adjust spacing and delimiters |
| Timestamp parse failure | time_format doesn’t match extracted string | Verify time_format mirrors the timestamp exactly (%Y-%m-%d %H:%M:%S) |
| Missing fields in output | Named capture groups incomplete or greedy | Check capture group boundaries; use character classes like [^\]]+ instead of .* |
| fluent.warn floods logs | Many unmatched records emit warnings | Add emit_invalid_record_to_error false in filter block |
| Config ignored after restart | Syntax error or option in wrong directive | Check td-agent.log for config parse errors; move emit_invalid_record_to_error to filter |
Fluentd Log Parser Performance, CPU Optimization, and Scaling Considerations
Regular expression parsing eats way more CPU than JSON deserialization. Backtracking in complex patterns makes it worse. Expressions like .* followed by specific captures can cause catastrophic backtracking when there’s no match. Keep regex patterns as specific as you can: use [^\]]+ instead of .*? when you know the delimiter. Limit captured fields to what you actually need for routing and indexing. Every named group adds extraction overhead.
Structured logging with JSON kills regex cost entirely. If you control the application, emit JSON and use @type json for near-instant parsing. When regex is your only option, test patterns with representative samples and measure CPU impact under expected log volume. Pre-compile and validate patterns with a Ruby regex tester to catch greediness and verify deterministic performance. For high-throughput pipelines, consider offloading parsing to a dedicated Fluentd aggregator tier or using faster parsers (json, msgpack) wherever you can.
Best practices for high-volume log pipelines:
- Prefer structured JSON logs over unstructured text to cut parsing CPU
- Use non-backtracking character classes like
[^\s]+or[^']+instead of greedy.* - Limit the number of named capture groups to required fields only
- Test regex patterns against worst-case input to catch catastrophic backtracking
- Dedicate separate Fluentd instances or aggregator nodes for CPU-intensive parsing tasks
Final Words
You’ve got the essentials to parse logs: core parser settings, plugin choices (regexp, json, apache/nginx, syslog), concrete config snippets, multiline handling, and troubleshooting tips.
Key reminders: use named captures, set timekey/timeformat, restart Fluentd after config changes, watch for “pattern not matched” warnings, and prefer JSON when parsing performance matters.
Apply these steps to make your pipeline reliable, tweak regexes, validate multiline rules, and monitor CPU. With a tuned fluentd log parser you’ll catch structured fields cleanly and avoid late-night alerts.
FAQ
Q: What does a Fluentd log parser do and when should I apply it?
A: A Fluentd log parser extracts structured fields from raw log text and should be applied at ingestion (source or parser filter) so logs are transformed for routing, indexing, and downstream processing.
Q: What are the key parser options like keyname, reservedata, @type regexp, and named captures?
A: Key parser options include keyname (input field to parse), reservedata (keep original), @type regexp (use Ruby‑style regex engine), and named captures (?
Q: How does Fluentd extract timestamps and handle unmatched logs?
A: Timestamp extraction uses timekey and timeformat (for example %Y-%m-%d %H:%M:%S). Unmatched logs emit “pattern not matched” warnings; suppress with emitinvalidrecordtoerror false in a filter.
Q: Which parser should I use for regex, JSON, Apache/Nginx, or syslog formats?
A: Use regex for custom or irregular formats, JSON for structured logs and better performance, Apache/Nginx parsers for common access logs, and syslog parser for rfc3164/rfc5424 messages.
Q: How do named capture groups map to Fluentd fields?
A: Named capture groups like (?
Q: How do I parse multiline logs and stack traces with Fluentd?
A: For multiline logs use format_firstline to detect event boundaries and continuation patterns (format1, format2). That stores multi‑row events as one field, ideal for stack traces.
Q: What causes “pattern not matched” and how do I debug parser errors?
A: “Pattern not matched” occurs when delimiters, spacing, or timestamp formats don’t match your regex. Debug with Fluentd debug logs, test regex against samples, and restart after config changes.
Q: What parser settings must every new Fluentd user configure?
A: The five required settings are @type (parser type), keyname, timekey, timeformat, and reservedata. Set these to ensure reliable field extraction and timestamp handling.
Q: How should I test and validate my Fluentd parser configuration?
A: Test by pasting representative logs, running Fluentd in debug, checking for “pattern not matched” warnings, verifying named fields in output, and iterating regex/time_format fixes.
Q: How do regex performance and scaling affect parser choice?
A: Regex parsing is CPU‑intensive and can backtrack; prefer JSON for high throughput, keep patterns specific, avoid nested quantifiers, and scale with more workers or sharded inputs.