Still scraping text logs with grep and hoping for the best?
Plain-text logs hide context and eat hours during incidents.
This post shows how to swap your default formatter for structured output in Python, Java, and Node.js, and why that change pays off—searchable fields, consistent timestamps, and reliable request correlation.
You’ll get minimal code examples, config tips for JSON vs logfmt, and a short checklist for field names and ingestion quirks so your logs actually work with real monitoring pipelines.
Practical Implementation of Structured Log Formatting
Most production logging frameworks ship with unstructured plaintext defaults. You’ll need to swap in a formatter that serializes log records to JSON or logfmt on every call.
Python’s standard library logging module lets you replace the default Formatter class with one that spits out JSON. Here’s a minimal example using python-json-logger:
import logging
from pythonjsonlogger import jsonlogger
logger = logging.getLogger()
handler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter('%(timestamp)s %(level)s %(name)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.info('Payment processed', extra={'request_id': 'abc-123', 'amount': 29.99})
# Output: {"timestamp": "2026-01-30T14:23:11.234Z", "level": "INFO", "name": "root", "message": "Payment processed", "request_id": "abc-123", "amount": 29.99}
In Go, zap gives you production-ready structured logging out of the box. Initialize a JSON logger like this:
package main
import "go.uber.org/zap"
func main() {
logger, _ := zap.NewProduction()
defer logger.Sync()
logger.Info("Payment processed",
zap.String("request_id", "abc-123"),
zap.Float64("amount", 29.99))
}
// Output: {"level":"info","ts":1738249391.234,"msg":"Payment processed","request_id":"abc-123","amount":29.99}
Java applications using Logback can switch to JSON with the logstash-logback-encoder library. Drop this encoder into your logback.xml:
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
When you log from Java code with logger.info("Payment processed", kv("request_id", "abc-123"), kv("amount", 29.99)), the encoder writes JSON to stdout.
Enabling structured formatting in any language:
- Install or configure a structured logging library or formatter targeting JSON or logfmt output.
- Replace your default logging handler or appender with one using the structured formatter.
- Define base fields your org will log in every event. At minimum: timestamp, level, service name, message, and request ID.
- Pass contextual metadata (user ID, correlation ID, environment) as structured fields, not concatenated strings inside the message.
- Write logs to stdout or stderr in containerized environments so orchestrators can collect them without filesystem dependencies.
Configuring and Customizing Logging Formatters
Structured formatters accept configuration options controlling timestamp precision, field ordering, indentation, and which metadata appears in every log entry. These settings directly affect downstream parsing, indexing performance, and whether alerts fire correctly.
In Python’s structlog, you define a processor chain that builds the final dictionary. Common processors add ISO 8601 timestamps, thread IDs, and stack traces on exceptions. You can inject a correlation ID processor that pulls the active request context and adds request_id to every log line emitted during that request’s lifecycle. The final processor serializes the dictionary to JSON or logfmt.
Node.js libraries like Winston allow similar chaining. You configure a format pipeline that timestamps, adds default fields, merges custom metadata, and then serializes. In Java, Logback’s LogstashEncoder exposes properties for timestamp format, field names, and whether to include stack traces as nested objects or flattened strings.
Field naming consistency matters more than the exact names you pick. If one service logs requestId and another logs request_id, your aggregation queries must union both names, and correlation breaks. Document a schema once (timestamp, level, message, requestid, traceid, user_id) and enforce it via shared libraries or linters.
Common configuration fields to standardize:
timestamp: Use ISO 8601 with UTC timezone (example: 2026-01-30T14:23:11.234Z). Avoid epoch milliseconds unless your ingestion tool requires them.
level: Map standard severity names (DEBUG, INFO, WARN, ERROR, CRITICAL) to a single field. Some tools expect lowercase strings, others expect integers.
message: A human-readable string summarizing the event. Keep it short. Put details in structured fields.
request_id: A unique identifier generated at request entry that propagates through all logs for that transaction. Essential for tracing a single API call across services.
correlationid or traceid: Distributed tracing identifier, often generated by an APM agent or ingress proxy. Lets you correlate logs with trace spans.
custom metadata: Application-specific fields like user_id, order_id, environment, service_name, host, process_id. Index only the fields you’ll query. Store the rest as unindexed text to control cost.
Comparing Structured Logging Formats
JSON and logfmt are the two dominant formats for structured logs. JSON nests objects and arrays, supports rich types (numbers, booleans, nulls), and integrates natively with Elasticsearch, cloud log stores, and most ingestion pipelines. logfmt flattens all fields into key=value pairs separated by spaces, making it trivially parseable with awk and grep, but you lose hierarchy.
Performance differences show up at high throughput. JSON serialization in Python and Node.js allocates temporary strings and dictionaries. At 10,000 logs per second, CPU overhead becomes measurable. logfmt serialization is faster because it skips nested encoding, but the trade-off is losing nested context like error.stack_trace or http.request.headers.
If your infrastructure already runs JSON pipelines (Fluentd, Logstash, Filebeat), adding a custom logfmt parser introduces maintenance cost. If you’re shipping logs from a constrained environment (edge devices, serverless functions with tight CPU budgets), logfmt’s lower overhead can matter.
| Format | Strengths | Weaknesses | Ecosystem Support |
|---|---|---|---|
| JSON | Rich nesting, wide tooling support, native in Elasticsearch/Kibana, cloud log services | Larger payload size, higher serialization CPU cost, verbose when pretty-printed | Elasticsearch, Splunk, Datadog, Grafana Loki, CloudWatch, GCP Logging, Azure Monitor |
| logfmt | Compact, fast to serialize, trivial command-line parsing (grep/awk), human-readable in terminals | No nesting, no arrays, harder to represent complex objects, less common in enterprise tools | Heroku, Prometheus text format, some Kubernetes controllers, custom parsers |
| CSV | Simple, trivial to import into spreadsheets or SQL databases | Fixed schema required upfront, no nesting, fragile when messages contain commas or newlines | Legacy batch systems, ad-hoc analysis scripts, spreadsheet tools |
| Key-value (custom delimiters) | Flexible delimiter choice, simple to parse with regex, low overhead | No standard, every system defines its own delimiter and escape rules, poor tooling | Custom log shippers, niche monitoring tools, internal scripts |
Choose JSON when you need to log nested request payloads, stack traces with frame details, or when your observability stack expects JSON by default. Choose logfmt when you’re optimizing for log volume, running on resource-constrained hosts, or you want human-readable logs you can grep in real time without parsing overhead. Avoid CSV and custom key-value formats unless you’re integrating with legacy systems that mandate them.
Integrating Structured Logs with Log Management Platforms
Log aggregation platforms expect predictable formats. Elasticsearch parses JSON fields into a schema at ingestion, creating indexed fields for fast queries. If your logs emit a mix of requestId and request_id, Elasticsearch creates two separate fields, and your dashboards break.
Most cloud providers (AWS CloudWatch, GCP Cloud Logging, Azure Monitor) accept newline-delimited JSON (NDJSON) on their ingestion APIs. Each log event is one JSON object per line with no comma separators.
Timestamp precision matters. Elasticsearch defaults to millisecond precision. Nanosecond timestamps require explicit mapping. If your formatter emits 2026-01-30T14:23:11.234567890Z but your ingestion mapping expects milliseconds, the extra digits are truncated or cause parsing errors. Check your platform’s documentation and align your formatter’s timestamp output.
Common ingestion expectations your structured logs must meet:
Newline-delimited JSON (NDJSON): Each log event is a single JSON object followed by a newline, with no surrounding array brackets or commas. This allows streaming ingestion without buffering the entire batch.
Timestamp field recognized by the platform: Elasticsearch looks for @timestamp by default. GCP Logging uses timestamp. Splunk auto-detects time fields but prefers time. Configure your formatter to use the expected field name or remap at ingestion.
Consistent field names across all services: If service A logs http_status_code and service B logs status_code, aggregating “all 500 errors” requires two queries. Define a schema once and enforce it.
Stable severity levels: Map your log levels to a fixed set (DEBUG, INFO, WARN, ERROR, CRITICAL) and avoid custom levels like “VERBOSE” or “TRACE” unless your platform supports them. Some systems map levels to numeric values. Document that mapping.
Schema alignment with index templates: If you’re sending logs to Elasticsearch, define an index template that declares field types (keyword vs text, integer vs float). Dynamic mapping can infer types incorrectly. Strings containing only digits become integers, breaking queries.
Platforms like Grafana Loki and AWS CloudWatch Insights parse JSON fields automatically, but you still need to decide which fields to index. High-cardinality fields (full URLs with query strings, unique user emails) explode index size and slow queries. Store them as unindexed text or drop them at ingestion.
Best Practices for Reliable Structured Logging
Schema stability prevents alert fatigue and broken dashboards. When you add a new field, existing queries continue to work. When you rename a field, every dashboard and alert that references the old name breaks.
Version your log schema and deprecate fields slowly. Log both old and new names for one release cycle before removing the old one.
Don’t embed large objects as strings. If you log json.dumps(request.body) into a message field, downstream tools see an escaped JSON string, not structured fields. Instead, log body_size_bytes and key extracted fields (content_type, user_id) as top-level fields. Store the full payload only when debugging specific incidents, and sample it at 1% to control volume.
Performance degrades when formatters execute expensive operations on every log call. JSON serialization of large dictionaries, deep stack trace formatting, and timestamp parsing with complex timezone logic all add microseconds per log line. At 10,000 logs per second, those microseconds become 10 to 100ms of CPU per second.
Profile your logging hot path in load tests and optimize serialization. Precompute static fields, reuse formatter instances, and defer expensive computations until logs are actually written (not when log level is below threshold).
In distributed systems, every log entry should include service_name, environment (prod/staging/dev), and host or pod_id. This metadata costs a few extra bytes per log but eliminates ambiguity when multiple services emit the same error message. You’ll answer “which service threw this error?” in one query instead of guessing.
Final Words
We implemented JSON-formatted structured logs in Python, Go, and Java, walked through formatter configuration, compared JSON vs logfmt, and showed how to feed logs into common ingestion systems. You also got a numbered checklist for enabling structured formatting and practical tips for field naming and NDJSON.
Follow the best practices for schema stability, request IDs, and compact payloads so searches stay fast and costs stay predictable.
Try the structured logging formatter in a staging run – you’ll spot issues fast and ship with clearer logs.
FAQ
Q: What is structured log formatting and why should I use it?
A: Structured log formatting is emitting logs as machine-readable objects (JSON or logfmt) to make indexing, searching, alerting, and automated parsing reliable across services and tools.
Q: What core fields should structured logs include?
A: Structured logs should include timestamp, severity, message, requestid (or traceid), service name, and contextual metadata to enable tracing, filtering, and reliable indexing in log systems.
Q: How do I enable JSON-formatted structured logs in Python, Go, and Java?
A: To enable JSON-formatted structured logs in Python, Go, and Java, choose a JSON-capable library (structlog/zap/Logback JSON), attach contextual fields to the logger, and switch the formatter/encoder to JSON with minimal config.
Q: How do I customize formatter fields like timestamp format or key ordering?
A: You customize formatter fields by configuring your library’s encoder/formatter options: set timestamp format, compact versus pretty JSON, field names/order, and add or remove metadata to match your parser expectations.
Q: JSON vs logfmt — which should I choose?
A: Choose JSON for rich, hierarchical data and broad tool support; choose logfmt for compact, human-friendly lines and fast CLI parsing. Pick JSON if you need complex querying or integrations.
Q: What ingestion expectations do log management platforms have?
A: Log management platforms typically expect NDJSON or newline-delimited records, precise timestamps, consistent field names, stable severity values, and schema alignment so indexing and alerts work reliably.
Q: What are best practices for reliable structured logging?
A: Best practices for reliable structured logging include keeping a stable schema, avoiding giant stack traces in one field, using consistent key names, embedding request/correlation IDs, and monitoring logging performance.
Q: How should I add request IDs and correlation IDs to logs?
A: Add request and correlation IDs by propagating them in request context or middleware, attaching them to the logger’s context for every log, and using a consistent key name across services.
Q: Will structured logging impact performance and how do I mitigate it?
A: Structured logging can add serialization cost; mitigate by using compact JSON, async or batched logging, sampling verbose logs, and avoiding expensive object serialization inside hot paths.