Logstash Message Format: Structure and Configuration Essentials

If your Logstash input is still dumping raw text into message fields, you’re wasting CPU and creating debugging headaches.
This post strips the noise and shows the logstash message format: which fields matter (@timestamp, message, host, tags), which codecs to use (json, json_lines, plain, multiline), and why the right choice saves parsing time and index pain.
You’ll get practical config snippets, common gotchas (multiline, charset, filter order), and a quick checklist to get structured events into Elasticsearch in minutes.

Core Concepts Behind Structuring the Logstash Message Format

A Logstash message is basically an event object made up of fields that represent one record moving through your pipeline. Every event has required and optional fields that determine how downstream systems index, search, and analyze your log data. The two critical fields are @timestamp, which marks when the event happened, and message, which contains the raw log text before you parse it. You’ll also see common fields like host (the originating machine), tags (labels your filters or inputs apply), and type (a category identifier you set during ingestion).

JSON is the go-to message format. It gives you predictable key/value extraction without burning CPU cycles on heavy parsing. When logs arrive as JSON objects, Logstash can map each key straight into an event field using the json codec. A JSON message like {"timestamp":"2026-03-25T12:34:56Z","level":"INFO","message":"User logged in","user_id":1234,"source_ip":"10.0.0.1"} becomes structured fields ready for filtering and indexing right away. Unstructured logs, plain-text syslog or custom application output, need grok patterns or dissect rules to extract meaning. That adds CPU cost and complexity.

Logstash handles structured and unstructured logs differently at ingestion. Structured formats (JSON, CEF, key/value pairs) skip expensive parsing steps and enter the filter stage with fields already populated. Unstructured formats land in the message field as raw text and depend on filter plugins to carve out individual fields. The @timestamp field gets auto-set by Logstash at ingestion time unless you override it with the date filter, which parses a timestamp string from your log using patterns like ISO8601 or custom formats such as dd/MMM/yyyy:HH:mm:ss Z.

Core Logstash event fields you’ll work with:

• @timestamp — The canonical timestamp for the event, typically set by the date filter or defaulted to ingestion time.
• message — The raw log line or original text content. This field often gets parsed and then either discarded or kept for audit purposes.
• host — The source hostname or IP address, usually populated by your input plugin or extracted from the log itself.
• tags — An array of labels (like _grokparsefailure) added by inputs, filters, or conditionals to flag special conditions or routing criteria.
• type — A classification label you set via input config (such as type => "apache") to simplify downstream filter logic and output routing.

Understanding JSON and Other Structured Logstash Message Formats

JSON is the default choice for structured logging because codecs like json and json_lines automatically parse each field into the Logstash event without extra filter config. When you set codec => json on an input, every incoming line is expected to be a complete JSON object, and Logstash maps each key into an event field instantly. The json_lines codec handles line-delimited JSON streams, common over TCP connections, where each line is an independent JSON object. Both codecs eliminate the need for grok patterns and reduce pipeline CPU usage, making JSON perfect for high-throughput environments where logs get emitted directly from applications in structured form.

Common Event Format (CEF) and syslog (RFC3164, RFC5424) offer alternative structured formats widely used in security and network infrastructure logging. CEF messages follow a key/value structure with predefined fields such as cs1 (custom string 1), src, dst, sev (severity), and rt (receipt time). Logstash includes a CEF codec that automatically extracts these fields into the event. Syslog formats include built-in timestamp, hostname, program name, and message components. Logstash can consume syslog via the syslog input plugin or parse it manually with grok patterns. RFC3164 is the older, widely supported syslog standard. RFC5424 adds structured data elements and UTF-8 encoding.

Codecs and Their Role in Shaping the Logstash Message Format

Codecs sit at the input and output boundaries of your Logstash pipeline. They translate raw bytes into event objects on ingestion and serialize events back into bytes at output. The json codec parses a complete JSON object per input record, mapping each key into an event field automatically. This codec is your first choice when your application logs are already emitted as JSON objects, since it skips the CPU cost of regex parsing. The json_lines codec handles streams where each line is an independent JSON object, commonly used with TCP or HTTP inputs receiving continuous log streams from distributed services.

The plain codec treats every input line as raw text, placing the entire line into the message field without any field extraction. It accepts a charset parameter to handle encoding, such as codec => plain { charset => "UTF-8" }. This codec is the default when you don’t specify one and works well for unstructured logs that you’ll parse later with grok or dissect filters. The multiline codec tackles the challenge of multi-line log entries, such as Java stack traces or error messages spanning multiple lines. It stitches consecutive lines into a single event by matching a pattern and deciding whether to append lines to the previous event or to the next event.

Your codec choice directly controls how Logstash interprets incoming data and what structure each event assumes before entering the filter stage. Choosing the wrong codec, like applying plain to JSON logs, results in a single message field containing the entire JSON string. That forces you to add a json filter to re-parse it. Similarly, failing to use multiline for stack traces causes each line to become a separate event, breaking the logical grouping you need for debugging. The codec also influences character encoding handling: if your logs contain non-ASCII characters or byte-order marks (BOM), specifying the correct charset in the codec config ensures Logstash reads and decodes the data correctly.

Common codec examples with configuration snippets:

1. json codec — codec => json — Parses each input line as a complete JSON object; automatically maps all keys into event fields.
2. json_lines codec — codec => json_lines — Processes line-delimited JSON streams where each line is an independent JSON object, typical for TCP inputs.
3. plain codec with UTF-8 — codec => plain { charset => "UTF-8" } — Treats each line as raw text and places it in the message field; useful for unstructured logs requiring grok parsing.
4. multiline codec — codec => multiline { pattern => "^\s" negate => true what => "previous" } — Joins lines that don’t start with whitespace to the previous event, common for stack traces or multi-line error messages.

Parsing and Transforming the Logstash Message Format with Filters

Filters sit in the middle of your Logstash pipeline and reshape raw messages into structured, enriched events ready for indexing or forwarding. The two most common parsing filters are grok and dissect. Both extract fields from the message string but differ in performance and flexibility. The mutate filter handles type conversions, field renames, and value masking. The date filter normalizes timestamp strings into the canonical @timestamp field. Together, these filters transform unstructured or semi-structured logs into fully indexed, searchable events with consistent field names and types.

Filter order matters. A typical sequence runs grok or dissect first to extract fields, then mutate to rename or convert types, then date to set @timestamp, and finally any enrichment filters like geoip or useragent. If the date filter runs before grok extracts the timestamp field, Logstash won’t find the field and will skip the conversion. Similarly, if you try to convert a field to an integer before it’s extracted from the message, the conversion silently fails. Always test filter chains incrementally and inspect the event structure at each stage using stdout { codec => rubydebug } to confirm fields appear as expected.

Grok and Dissect Pattern Use Cases

Grok is the standard filter for parsing unstructured logs into structured fields using named regular expressions. It ships with over 100 built-in patterns covering common formats like Apache logs (%{COMBINEDAPACHELOG}), syslog (%{SYSLOGLINE}), and IP addresses (%{IP}). A grok filter matches the message field against a pattern and, when successful, creates a new field for each named capture group. For example, filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } extracts fields like clientip, verb, request, response, and bytes from an Apache access log line.

Dissect is a faster, fixed-field parser that works well when your logs follow a consistent, delimiter-separated structure. Instead of regular expressions, dissect uses a simple template with placeholders, such as dissect { mapping => { "message" => "%{timestamp} %{host} %{program}: %{msg}" } }, and splits the message on whitespace or other delimiters. Dissect is up to five times faster than grok but can’t handle variable-length fields or optional components. Use dissect for high-throughput pipelines with predictable log formats and grok for logs with variability or complex patterns.

Mutate Rules for Type Conversion and Redaction

The mutate filter performs field-level transformations including renaming, type conversion, and substring replacement. Common operations include rename to standardize field names across different log sources, like mutate { rename => { "source_ip" => "src_ip" } }, and convert to change field types from strings to integers, floats, or booleans, such as mutate { convert => { "bytes" => "integer" "response_time" => "float" } }. Type conversion is critical for numeric aggregations in Elasticsearch. Without it, fields like bytes remain strings and can’t be summed or averaged.

The gsub option within mutate lets you redact or sanitize sensitive data using regular expressions. For example, mutate { gsub => [ "message", "password=\w+", "password=****" ] } replaces any password value in the message field with asterisks. This operation modifies the field in place before the event reaches the output, ensuring sensitive data never lands in your index. Mutate can also add static fields, like mutate { add_field => { "environment" => "production" } }, or remove unwanted fields, like mutate { remove_field => [ "temporary_field" ] }.

Date Filter for Reliable Timestamp Normalization

The date filter parses a timestamp string from a named field and overwrites the @timestamp field with the parsed value. This is essential when logs contain their own timestamps that differ from the ingestion time. The filter accepts an array of patterns to try in order until one matches, such as date { match => [ "timestamp", "ISO8601", "dd/MMM/yyyy:HH:mm:ss Z" ] target => "@timestamp" }. If the first pattern fails, Logstash attempts the next, and so on. When all patterns fail, the date filter adds the _dateparsefailure tag and leaves @timestamp unchanged.

Timezone handling is a common source of errors. If your timestamp string lacks a timezone indicator, Logstash assumes UTC. To override this, set the timezone option, like date { match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ] timezone => "America/New_York" target => "@timestamp" }. Always emit timestamps in ISO8601 format or include explicit timezone offsets in your logs to avoid ambiguity. After the date filter runs, inspect @timestamp to confirm it reflects the correct time. Incorrect timestamps cause events to appear out of order or in the wrong time range in Kibana.

Common causes of parsing failures include:

• Wrong codec on input (such as treating JSON logs as plain text)
• Multiline events not joined correctly (stack traces split across multiple events)
• Timestamp format mismatch between the log and the date filter patterns
• sincedb preventing re-read of files during testing (use sincedb_path => "/dev/null" to force re-parsing)

Input Plugin Formats and How They Influence Logstash Message Structure

Input plugins define how raw data enters your Logstash pipeline and which codec interprets that data. The file input reads log files from disk and supports codecs like multiline to join multi-line entries before they enter the filter stage. File input tracks read positions using a sincedb file, which prevents re-reading lines on every Logstash restart. For testing, set sincedb_path => "/dev/null" to ignore the position tracker and force Logstash to re-read the entire file from the beginning.

The beats input receives events from Filebeat, Metricbeat, or other Elastic Beats agents. Beats events are pre-structured with fields like @timestamp, message, host.name, and agent.type, so you often skip the codec entirely or use plain to accept the Beats event as is. The beats input listens on a TCP port (commonly 5044) and expects the Lumberjack protocol, which includes built-in compression and encryption. When using beats input, your pipeline starts with structured data, reducing the parsing burden on Logstash filters.

TCP and UDP inputs handle network streams, often used for syslog forwarding or application logs sent over the network. For JSON streams, set codec => json_lines to parse each line as a separate JSON object, like input { tcp { port => 5000 codec => json_lines } }. For plain syslog, use the syslog codec or leave it as plain and parse with grok. TCP inputs support SSL/TLS encryption for secure transport, configured with ssl_enable => true and certificate paths. UDP is faster but unreliable. Use it only when occasional log loss is acceptable.

Common codec usage patterns on different inputs:

• File input with multiline — input { file { path => "/var/log/app.log" codec => multiline { pattern => "^\d{4}-\d{2}-\d{2}" negate => true what => "previous" } } } — Joins stack traces or error messages spanning multiple lines.
• Beats input with no codec — input { beats { port => 5044 } } — Accepts structured events from Beats agents; codec defaults to plain but Beats protocol handles framing internally.
• TCP input with json_lines — input { tcp { port => 5000 codec => json_lines } } — Parses line-delimited JSON streams from applications or log forwarders over TCP.

Output Formatting and the Logstash Message Format

Outputs serialize Logstash events and send them to external systems, with formatting controlled by codecs and plugin-specific options. The rubydebug codec is your most useful debugging tool: it prints a human-readable representation of the entire event structure, including all fields, types, and metadata. Use it with the stdout output during development, like output { stdout { codec => rubydebug } }, to inspect how filters have shaped each event before it reaches production outputs like Elasticsearch or Kafka.

Elasticsearch outputs rely on index templates and dynamic mapping to interpret field types and structure nested objects. When you send a JSON event to Elasticsearch, field names become mapping keys, and field values determine types (string, integer, date, geopoint). If a field arrives as a string but should be an integer, Elasticsearch mapping conflicts occur, causing indexing failures. To avoid this, use mutate filters to convert types in Logstash before output, or define explicit Elasticsearch index templates that enforce field types regardless of incoming data.

Metadata, Tags, and Field Naming in Logstash Message Formats

The @metadata field is a special structure that holds temporary data during pipeline processing but never gets indexed or sent to outputs. Use @metadata to store routing context, temporary flags, or intermediate values that guide filter logic without cluttering the final event, like mutate { add_field => { "[@metadata][source]" => "%{path}" } }. You can reference metadata in conditionals and output blocks, such as routing events to different Elasticsearch indices based on [@metadata][target]. Because metadata is ephemeral, it keeps your indexed documents lean and focused on real log data.

Tags are labels added to events by inputs, filters, or conditionals to flag conditions or processing states. The most common tag is _grokparsefailure, which Logstash automatically adds when a grok pattern fails to match. Other filters add tags on failure: _dateparsefailure for the date filter, _jsonparsefailure for JSON parsing errors. You can add custom tags using the add_tag option in filters or inputs, like mutate { add_tag => [ "audit_log" ] }, and then route tagged events to specific outputs with conditionals like if "audit_log" in [tags] { ... }.

Field naming best practices:

• Use lowercase with underscores (snake_case) for consistency: user_id, src_ip, response_time.
• Avoid dots in field names unless you intend to create nested objects in Elasticsearch (such as host.name becomes a nested host object with a name field).
• Reserve @ prefix for Logstash system fields like @timestamp and @metadata. Never create custom fields starting with @.
• Use consistent field names across all log sources (always src_ip, never mix source_ip and client_ip). This simplifies cross-log queries and dashboards.

Handling Multiline, Encoding, and Long Messages in Logstash Format Workflows

Multiline log entries, such as Java stack traces, Python tracebacks, or multi-line JSON, require special handling to group related lines into a single event. The multiline codec on the input plugin is the recommended approach. It uses a regular expression pattern to detect line boundaries and a what directive to decide whether to append lines to the previous event or to the next event. For example, codec => multiline { pattern => "^\s" negate => false what => "previous" } appends any line starting with whitespace to the previous event, which is typical for stack traces where continuation lines are indented.

Character encoding issues are a frequent source of parsing failures, especially when logs contain non-ASCII characters or byte-order marks (BOM). UTF-8 is the standard encoding for modern applications, but older systems may emit logs in ISO-8859-1, Windows-1252, or UTF-16. Specify the correct encoding in the codec or input config, like codec => plain { charset => "ISO-8859-1" }, to ensure Logstash decodes the bytes correctly. If a BOM appears at the start of a file, it can break the first line’s parsing. Some codecs strip the BOM automatically, but you may need to manually remove it or use a filter to discard the BOM bytes.

Newline handling varies by operating system: Unix systems use \n, Windows uses \r\n, and older Mac systems used \r. Most Logstash inputs auto-detect newline conventions, but mixed line endings in a single file can cause parsing issues. If your logs contain embedded newlines within a single logical message (common in JSON blobs or XML), use the multiline codec to define the message boundary, or emit logs as single-line JSON objects to avoid ambiguity. Long messages exceeding input buffer limits may be truncated or split. Increase the max_message_size or buffer_size parameter in the input plugin if you encounter partial events.

Practical Examples of Logstash Message Formatting

A well-formed JSON log message from a web application might look like this: {"timestamp":"2026-03-25T12:34:56Z","level":"INFO","message":"User logged in","user_id":1234,"source_ip":"10.0.0.1"}. To ingest this with Logstash, configure a file input with the json codec: input { file { path => "/var/log/webapp.log" start_position => "beginning" codec => json } }. The codec automatically parses each key into an event field, so the Logstash event immediately contains fields like timestamp, level, message, user_id, and source_ip. You then add a date filter to set @timestamp from the timestamp field: filter { date { match => [ "timestamp", "ISO8601" ] target => "@timestamp" } }. Finally, output to Elasticsearch with an index pattern: output { elasticsearch { hosts => ["localhost:9200"] index => "webapp-%{+YYYY.MM.dd}" } }.

Syslog messages follow a predictable structure with a priority code, timestamp, hostname, program name, and message. An example RFC3164 syslog line: <34>Apr 15 10:22:33 webserver nginx: 192.168.1.100 GET /index.html 200 1234. To parse this, use the syslog input plugin: input { syslog { port => 514 } }, which automatically extracts fields like syslog_timestamp, syslog_hostname, syslog_program, and syslog_message. If you’re reading syslog from a file instead of a network port, use the file input with a grok pattern: filter { grok { match => { "message" => "%{SYSLOGLINE}" } } date { match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] target => "@timestamp" } }. This approach works for both RFC3164 and RFC5424 formats, though RFC5424 requires a different pattern or the rfc5424_support option in the syslog input.

Apache and Nginx access logs are common targets for Logstash parsing. Apache’s Combined Log Format includes client IP, timestamp, HTTP method, request path, response code, and bytes sent. A sample line: 192.168.1.100 - - [15/Apr/2025:10:22:33 +0000] "GET /index.html HTTP/1.1" 200 1234 "-" "Mozilla/5.0". Parse it with the built-in grok pattern: filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] target => "@timestamp" } geoip { source => "clientip" } }. The grok pattern extracts fields like clientip, verb, request, response, and bytes, and the geoip filter adds geolocation data for the client IP. Nginx JSON logs are simpler: configure Nginx to emit access logs as JSON, then use codec => json on the file input to parse them directly without any filter plugins.

Troubleshooting and Debugging Logstash Message Format Issues

Testing your Logstash config before deploying to production catches syntax errors and logical mistakes early. Run bin/logstash --config.test_and_exit -f config.conf to validate the pipeline syntax without starting the service. Logstash checks for missing braces, invalid plugin options, and unsupported parameter values, printing errors to the console. If the test passes, the pipeline is syntactically correct, though it may still fail at runtime due to data mismatches or missing resources.

When events aren’t parsing as expected, increase Logstash’s log verbosity to see detailed processing information. Run bin/logstash -f config.conf --log.level debug to output debug messages showing filter matches, plugin execution, and field transformations. Add a stdout output with codec => rubydebug to print each event after filters run: output { stdout { codec => rubydebug } }. This shows the exact structure and field values Logstash created, making it easy to spot missing fields, wrong types, or unexpected values. Look for tags like _grokparsefailure, _dateparsefailure, or _jsonparsefailure, which indicate where parsing broke down.

Troubleshooting steps for common Logstash message format issues:

1. Confirm input codec matches message format — If logs are JSON, use codec => json; if plain text, use codec => plain and add grok filters. Wrong codec choice leaves data unparsed in the message field.
2. Temporarily enable stdout rubydebug — Add output { stdout { codec => rubydebug } } to inspect the raw event structure and confirm filters are extracting fields correctly.
3. Check for parsing failure tags — Look for _grokparsefailure, _dateparsefailure, or _jsonparsefailure in the tags array. These indicate which filter failed and where to refine patterns or formats.
4. Validate timestamps with the date filter — Ensure the @timestamp field reflects the correct time. If it doesn’t, the date filter pattern may not match your timestamp format or the field name is wrong.
5. Re-run with config test and debug logging — Use --config.test_and_exit to catch syntax errors, then --log.level debug to trace pipeline execution step by step. For file inputs, set sincedb_path => "/dev/null" to force Logstash to re-read files from the beginning during testing.

Final Words

In the action, we covered the essentials: core fields like @timestamp and message, JSON vs. syslog/CEF, codecs and multiline handling, filters for parsing and date normalization, inputs and outputs, metadata and naming, plus hands-on examples.

You also got practical debugging tips—use rubydebug, test configs, and watch for grok failures, encoding issues, and timestamp mismatches.

Apply these checks to shape a reliable logstash message format and cut down on surprises downstream. You’re set to iterate faster and ship cleaner logs.

FAQ

Q: What are the core fields in a Logstash message?

A: The core fields in a Logstash message are @timestamp, message, host, tags, type, plus any structured fields parsed by codecs or filters; JSON is commonly used for predictable key/value extraction and indexing.

Q: What is the @timestamp field and how do I set it?

A: The @timestamp field is the canonical event time; set or normalize it with the date filter using patterns like ISO8601 or custom formats to ensure correct time-based indexing and queries.

Q: How does Logstash treat structured versus unstructured logs?

A: Logstash treats structured logs (JSON, CEF) as parseable key/value events via codecs, while unstructured logs stay in the message field until filters like grok or dissect extract structured fields.

Q: Which message formats does Logstash commonly support and how are they parsed?

A: Logstash commonly supports JSON (key/value), CEF (security fields like severity, src, dst), and Syslog (RFC3164/RFC5424 with timestamp, host, program); each is parsed with its codec or appropriate grok/dissect patterns.

Q: When should I use json, json_lines, plain, or multiline codecs?

A: Use json or jsonlines for JSON payloads (jsonlines for line-delimited streams), plain for raw text, and multiline to join stack traces; always set charset (UTF-8) when needed to avoid encoding issues.

Q: How do grok and dissect patterns differ and what causes parsing failures?

A: Grok and dissect parse logs differently: grok uses regex-based patterns for flexibility, dissect is faster for fixed fields; failures usually come from pattern mismatches, wrong charset, unexpected multiline, or malformed input.

Q: What do mutate and date filters do in message shaping?

A: Mutate rules rename fields, convert types, and redact values; the date filter parses a timestamp into @timestamp using patterns like ISO8601 or dd/MMM/yyyy:HH:mm:ss Z to normalize event times.

Q: How do input plugins like file, beats, TCP, and UDP influence message structure?

A: Input plugins shape messages at ingestion: file uses sincedb and codecs for multiline, Beats sends structured events from Filebeat, and TCP/UDP often use json_lines for streaming—charset matters for correct decoding.

Q: How do outputs and index mappings affect final message storage?

A: Output plugins format events on the way out; stdout rubydebug shows full structure, while Elasticsearch obeys index templates and mappings, so field names and types determine storage, searchability, and mapping conflicts.

Q: What are best practices for @metadata, tags, and field naming?

A: Use @metadata for non-indexed routing data and tags to flag conditions like _grokparsefailure; name fields consistently, avoid spaces, prefer dot notation for nesting, and keep predictable keys for mappings.

Q: How should I handle multiline logs, encodings, and long messages?

A: Handle multiline with the multiline codec to stitch stack traces, ensure correct encoding (UTF-8/UTF-16) and strip BOMs if present; mismatched newlines or wrong codec settings often break parsing.

Q: What quick troubleshooting steps help fix message format issues?

A: Quick troubleshooting steps: test configs with bin/logstash –config.testandexit, run with –log.level debug, inspect events via stdout { codec => rubydebug }, refine patterns in Grok Debugger, and reset sincedb_path for file replays.