Python Logging Custom Formatter: How to Format Your Logs

Are your logs a mess that hides real bugs and noise?
A Python logging custom formatter fixes that by letting you control every field, order, and output shape.
In this post you’ll learn how to subclass logging.Formatter, format timestamps, emit JSON for aggregators, add colors for console, and attach context like request_id.
I’ll also cover performance gotchas and production best practices so your formatter is fast and safe.
By the end you’ll be able to write compact console output and structured logs that play nice with ELK or Splunk.

Understanding How a Python Logging Custom Formatter Works

A custom formatter gives you full control over how log lines look. You do this by subclassing logging.Formatter and overriding format(record). That method gets a LogRecord and must return the final string the handler will write. Inside format you can reorder fields, add computed values, apply colors, or emit JSON instead of plain text.

LogRecord exposes a lot of useful attributes: levelname for severity, the timestamp after you call self.formatTime(record), message for the rendered text, pathname and funcName for where the event happened, and exc_info when an exception is present. Pick and order the fields you want developers or operators to see, and you control the story each log line tells.

People write custom formatters for a few clear reasons. You want readable console output while debugging. You need structured JSON for aggregators like ELK or Splunk in production. Or you want different looks for console and file output so one stays compact and the other keeps full context.

Core formatter capabilities you’ll likely use:

• Timestamp formatting and timezone control.
• Injecting fields like module, line, process id, thread, or custom extras.
• Emitting structured output such as JSON or key=value pairs.
• Attaching different formats to different handlers.
• Customizing how exceptions and tracebacks appear.

Building a Custom Python Logging Formatter Class

Import logging, subclass logging.Formatter, and override format(self, record). The record is a LogRecord with attributes like levelname, name, lineno, and msg. Use record.getMessage() to get the formatted message text, or access record.dict for extras.

If you need a timestamp, call self.formatTime(record, datefmt) once and save it in a variable. Combine timestamp, level, message, and any context into a single string and return it. Use getattr(record, ‘funcName’, ‘unknown’) or record.dict.get(‘request_id’, ”) for fields that may be missing so your formatter never crashes.

Keep format lean. format runs for every log event. Avoid heavy work like large JSON serialization or expensive regex in the hot path. If you compute derived fields, cache them on the record the first time so subsequent handlers can reuse them.

Defining format(self, record) in a nutshell:

• Extract what you need: timestamp = self.formatTime(record, ‘%Y-%m-%d %H:%M:%S’), level = record.levelname, msg = record.getMessage().
• Safely include optional fields with getattr or record.dict.get.
• If record.excinfo exists, append self.formatException(record.excinfo).
• Return the final string. If anything goes wrong, catch exceptions and return a safe fallback so logging itself never brings the app down.

Applying a Custom Formatter to Console and File Handlers

Create an instance of your formatter and attach it via handler.setFormatter(your_formatter). For console, use logging.StreamHandler, usually with level logging.DEBUG so you see everything locally. For files, use logging.FileHandler or TimedRotatingFileHandler and often set the handler level to logging.INFO to avoid writing debug noise to disk.

Use different formatter instances when you want different shapes for console and file output. Disable duplicate entries by setting logger.propagate = False on custom loggers, and don’t add handlers repeatedly during reloads—check logger.handlers before calling addHandler.

Implementing Colorized Console Output

Colorized formatters wrap parts of the message in ANSI codes so the terminal shows colors. Map record.levelno to codes such as \033[31m for red and end with \033[0m to reset. For Windows compatibility, use colorama which gives Fore.RED and Style.RESET_ALL that work across platforms.

Align level names to a fixed width with record.levelname.ljust(8) so lines line up and are easier to scan. A simple layout that works well is: timestamp, aligned and colored level, then message. Example: “2025-01-15 10:32:45 [INFO ] Server started on port 8080”.

Example color choices:

• DEBUG: dim gray \033[90m
• INFO: green \033[32m
• WARNING: yellow \033[33m
• ERROR: red \033[31m
• CRITICAL: bright or bold red \033[91m or \033[1;31m

Creating JSON Structured Logs

Send JSON when you need logs that aggregators can parse. In format(self, record) build a dict with fields you care about, convert values to JSON-serializable types, then return json.dumps(record_dict). For streaming, write one JSON object per line.

Start by copying selected attributes from record.dict. Typical keys: timestamp, level, message, name, pathname, lineno, plus any extras like requestid or traceid. Convert timestamps to an ISO8601 string and omit internal fields that add noise.

Handle exceptions by checking record.excinfo and adding a formatted traceback string under “exception” or “stacktrace”. If exc_info is None, leave that key out.

Record to dict example:
log_entry = {
‘timestamp’: self.formatTime(record, ‘%Y-%m-%dT%H:%M:%S’),
‘level’: record.levelname,
‘message’: record.getMessage()
}
Then merge extras from record.dict excluding internal keys.

Keep the JSON compact. Drop fields like created, msecs, relativeCreated, and other internal keys unless they provide real value to your operators.

Adding Context: Filters and Adapters

To inject dynamic context like request ids or trace ids, use a Filter that sets record.context_id from contextvars, or use logging.LoggerAdapter to attach an extra dict at log creation time. Filters keep context injection separate from formatting and are reusable. Adapters work when you can pass a wrapped logger into the code that emits logs.

For exceptions, format tracebacks selectively. Some teams include full tracebacks for errors and omit them for warnings. Others store type and message only to keep JSON compact.

Common contextual fields to include:

• request_id for tying related logs together
• user_id for audits
• module and lineno for origin
• traceid and spanid for distributed tracing

Timestamps and Timezones

Control timestamp formatting via formatTime or the Formatter constructor datefmt. ISO8601 with microseconds is a solid default: ‘%Y-%m-%dT%H:%M:%S.%f’ gives precise timestamps that play well with traces and metrics.

logging uses local time by default. To switch to UTC globally, set logging.Formatter.converter = time.gmtime before creating formatters. For per-formatter control, override formatTime and use datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat() to produce a timezone aware string.

If you’re logging at high throughput, be mindful that timezone conversion can cost CPU. Cache formatted timestamps when many records fall in the same second, or choose a cheaper format after profiling.

Performance Tips

format() runs on the logging thread. Expensive work here slows your app. Avoid large json.dumps() on huge objects and skip expensive regex operations. If serialization is necessary, do it lazily and cache the result on the record.

Use f-strings or prepared templates for fast string assembly. Call self.formatTime(record) once and reuse the result. Keep the number of handlers and filters modest so each record travels through a short pipeline.

Key optimizations:

• Cache formatted timestamps for records in the same second
• Use simple string templates instead of many concatenations
• Avoid expensive functions inside format
• Extract fields directly instead of parsing messages with regex
• Reduce handler/filter count when possible

Production Best Practices

Never log secrets or sensitive personal data unless you have explicit redaction. Add a sanitization step that replaces tokens, emails, or credit card patterns before output. Some teams filter these fields out in a logging.Filter so the formatter never sees them.

Use INFO or higher for file logs in production. Keep console at DEBUG for local work but avoid persisting debug noise. Rotate logs with TimedRotatingFileHandler or RotatingFileHandler so disk usage stays bounded.

Disable propagation on custom loggers when attaching handlers to avoid duplicate entries. Tune logger and handler levels so the logger captures the lowest level you might need and handlers filter down to the levels you actually want to persist.

Final Words

in the action we covered how a formatter shapes output, subclassing logging.Formatter and overriding format(), using record attributes, attaching different formatters to console and files, colorized and JSON formats, injecting request/trace ids, custom timestamps, and performance tradeoffs.

Keep format() light, handle missing fields, and pick per-environment formats—readable console for dev, structured ndjson for pipelines. Watch timezone and redaction.

A good python logging custom formatter cuts noise and speeds debugging. You’ll see clearer logs and fewer late-night surprises.

FAQ

Q: What is a Python logging custom formatter and what does it do?

A: The Python logging custom formatter controls how LogRecord objects are converted into strings by overriding format(), letting you reorder fields, add timestamps, and produce structured or human-friendly log output for handlers.

Q: How do I create a custom formatter class?

A: You create a custom formatter by subclassing logging.Formatter and overriding format(record); use formatTime for timestamps, handle missing attributes defensively, and avoid heavy work inside format for performance.

Q: How do I use record attributes to include module, function, and line information?

A: You use record attributes like record.levelname, record.pathname, record.funcName, record.lineno, and record.getMessage(); access record.dict for extras and assemble the final formatted string or JSON object.

Q: How do I add custom fields like requestid or traceid to logs?

A: You add custom fields by using LoggerAdapter or a Filter to inject extras (requestid, traceid) into record.dict, then read those keys in your formatter and fall back if missing.

Q: How do I attach a formatter to console and file handlers?

A: You attach a formatter with handler.setFormatter(my_formatter); use a color formatter for console (DEBUG), TimedRotatingFileHandler for files (INFO), and disable propagation to avoid duplicate entries.

Q: How do I implement a colorized console output formatter?

A: You implement colorized output by mapping record.levelno to ANSI color codes (or colorama), wrapping the message with codes, and using a compact layout like timestamp, aligned level, and message for clarity.

Q: How do I create JSON structured logs for ingestion?

A: You create JSON logs by converting the LogRecord to a dict (merge extras, strip internals), then json.dumps one object per line (NDJSON) so ELK/Graylog can consume logs reliably.

Q: How do I format timestamps with microseconds and handle timezones?

A: You format timestamps using formatTime(record, datefmt) with %f for microseconds and override Formatter.converter for timezone-aware timestamps, while being mindful of conversion performance and caching needs.

Q: What performance considerations should I follow for custom formatters?

A: You avoid expensive ops in format (heavy json.dumps, slow calls), use lazy or cached formatting, minimize string concatenation, and keep format() lightweight for high-throughput logging paths.

Q: What are best practices for production-ready logging formatters?

A: You redact sensitive fields, prefer JSON for ingestion, set file logs to INFO+, enable rotation, disable propagation to prevent duplicates, and align logger and handler levels for predictable output.

Q: How should format() handle missing fields and exceptions safely?

A: You handle missing fields with record.dict.get(‘key’, fallback), wrap formatting in try/except to avoid crashes, and include stack traces using formatException when record.exc_info is present.

Q: How do I configure custom formatters using dictConfig?

A: You configure custom formatters in dictConfig under ‘formatters’ (class, format, datefmt, style), reference them by name in handler entries, and attach handlers to loggers to apply formatters.