PyLogShield Logger¶
The main logger class that extends Python's standard logging.Logger with additional features.
Quick Reference¶
from pylogshield import get_logger, PyLogShield
# Recommended: Use get_logger for singleton pattern
logger = get_logger("my_app", log_level="INFO", enable_json=True)
# Alternative: Direct instantiation
logger = PyLogShield("my_app", log_level="INFO")
Common Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
name |
str |
Required | Logger name |
log_level |
str \| int |
INFO |
Logging level |
enable_json |
bool |
False |
Output JSON format |
use_queue |
bool |
False |
Async logging |
use_rich |
bool |
False |
Rich console output |
rate_limit_seconds |
float |
0.0 |
Rate limiting interval |
log_directory |
str \| Path |
~/.logs |
Log file directory |
log_file |
str |
{name}.log |
Log file name |
rotate_file |
bool |
False |
Enable log rotation |
rotate_max_bytes |
int |
5000000 |
Max file size before rotation |
rotate_backup_count |
int |
5 |
Number of backup files |
add_console |
bool |
True |
Add a console handler on creation |
enable_metrics |
bool |
False |
Enable metrics tracking |
enable_context_scrubber |
bool |
True |
Remove cloud credentials |
enable_context |
bool |
False |
Install ContextFilter; pairs with log_context()/async_log_context() |
queue_maxsize |
int |
0 |
Max async queue size (0 = unbounded); only used when use_queue=True |
flowchart TB
INIT(["PyLogShield(\n enable_json=True,\n use_queue=True,\n enable_context=True\n)"])
subgraph HANDLERS ["Configured Handlers"]
F["FileHandler / RotatingFileHandler\n(log_directory / rotate_file)"]
C["ConsoleHandler / RichHandler\n(add_console / use_rich)"]
J["JsonFormatter\n(enable_json=True)"]
Q["QueueHandler → QueueListener\n(use_queue=True, queue_maxsize)"]
end
subgraph FILTERS ["Active Filters"]
CF["ContextFilter\n(enable_context=True)"]
CS["ContextScrubber\n(enable_context_scrubber=True)"]
KF["KeywordFilter\n(log_filter=...)"]
end
subgraph FEATURES ["Runtime Features"]
RL["RateLimiter\n(rate_limit_seconds > 0)"]
MT["LogMetricsHandler\n(enable_metrics=True)"]
end
INIT --> HANDLERS
INIT --> FILTERS
INIT --> FEATURES
Examples¶
Basic Logging¶
from pylogshield import get_logger
logger = get_logger("my_app")
logger.debug("Debug message")
logger.info("Info message")
logger.warning("Warning message")
logger.error("Error message")
logger.critical("Critical message")
Sensitive Data Masking¶
logger = get_logger("secure_app")
# Enable masking with mask=True
logger.info({"user": "john", "password": "secret"}, mask=True)
# Output: {"user": "john", "password": "***"}
JSON Logging¶
logger = get_logger("json_app", enable_json=True)
logger.info("User logged in")
# Output: {"timestamp": "2024-01-15T10:30:00+00:00", "level": "INFO", ...}
Log Rotation¶
logger = get_logger(
"rotating_app",
rotate_file=True,
rotate_max_bytes=10_000_000, # 10 MB
rotate_backup_count=5
)
Context Propagation¶
from pylogshield import get_logger
from pylogshield.context import log_context
logger = get_logger("api", enable_context=True, enable_json=True)
with log_context(request_id="abc-123"):
logger.info("Processing")
# JSON output includes request_id field
Exception Logging with Masking¶
try:
connect_db(password=secret)
except Exception:
logger.exception("DB connection failed", mask=True)
# exception .args are masked; traceback locals are NOT redacted
Warning
mask=True does not redact traceback frame locals. See Sensitive Data Masking for details.
Replacing an Existing Logger (force=True)¶
import logging
# Third-party code may already have a standard logger registered
logging.getLogger("shared_service")
# get_logger raises TypeError by default — use force=True to replace it
logger = get_logger("shared_service", force=True, enable_json=True)
Warning
force=True emits a UserWarning before replacing the existing logger.
Any code that already holds a reference to the old logger will stop
receiving records once it is replaced.
Production Setup with Async + Rotation + Context¶
from pylogshield import get_logger, add_sensitive_fields
from pylogshield.context import log_context
# Register domain-specific sensitive fields once at startup
add_sensitive_fields(["account_number", "sort_code", "national_id"])
logger = get_logger(
"payments",
log_level="INFO",
enable_json=True, # Structured output for ELK / Datadog
rotate_file=True, # Rotate when file hits 50 MB
rotate_max_bytes=50_000_000,
rotate_backup_count=10,
use_queue=True, # Non-blocking: log calls return immediately
queue_maxsize=100_000, # Drop new messages if queue fills (not block)
rate_limit_seconds=1.0, # At most 1 identical message per second
enable_metrics=True, # Count logs by level
enable_context=True, # Allow log_context() injection
)
def process_payment(user_id: int, amount: float, account_number: str) -> None:
with log_context(user_id=user_id, operation="payment"):
logger.info(
{"account_number": account_number, "amount": amount},
mask=True, # account_number → "***"
)
# ... business logic ...
logger.info("Payment authorised")
# At application shutdown — flush remaining queued messages
logger.shutdown()
metrics = logger.get_metrics()
print(f"Logged {metrics['count']} records in {metrics['elapsed']:.1f}s")
API Reference¶
PyLogShield(name: str, *, log_level: Union[LogLevel, str, int] = logging.INFO, enable_json: bool = False, use_queue: bool = False, use_rich: bool = False, rate_limit_seconds: float = 0.0, log_directory: Union[str, Path, None] = None, log_file: Optional[str] = None, rotate_file: bool = False, rotate_max_bytes: int = 5000000, rotate_backup_count: int = 5, add_console: bool = True, enable_metrics: bool = False, log_filter: Optional[Union[logging.Filter, KeywordFilter, Iterable[str]]] = None, enable_context_scrubber: bool = True, enable_context: bool = False, queue_maxsize: int = 0)
¶
Bases: Logger
A structured logger with redaction, rate limiting, async support, and context scrubbing.
This logger extends Python's standard logging.Logger with features commonly needed in data engineering and production environments:
- Sensitive Data Masking: Automatically redact passwords, tokens, API keys, etc.
- Rate Limiting: Prevent log flooding from repetitive messages.
- JSON Formatting: Structured logging for log aggregation systems.
- Async Logging: Non-blocking logging via queue handlers.
- Rich Console: Colorized output for development environments.
- Context Scrubbing: Remove cloud credentials from log records.
- Metrics: Track log volume and rates.
| PARAMETER | DESCRIPTION |
|---|---|
|
The name of the logger.
TYPE:
|
|
The logging level. Default is logging.INFO.
TYPE:
|
|
Whether to output logs in JSON format. Default is False.
TYPE:
|
|
Whether to use async logging via queue handlers. Default is False.
TYPE:
|
|
Whether to use Rich library for colorized console output. Default is False.
TYPE:
|
|
Minimum seconds between identical log messages. Default is 0.0 (disabled).
TYPE:
|
|
Directory for log files. Default is ~/.logs.
TYPE:
|
|
Name of the log file. Default is "{name}.log".
TYPE:
|
|
Whether to enable log file rotation. Default is False.
TYPE:
|
|
Maximum file size before rotation. Default is 5,000,000 bytes.
TYPE:
|
|
Number of backup files to keep. Default is 5.
TYPE:
|
|
Whether to add a console handler. Default is True.
TYPE:
|
|
Whether to enable logging metrics. Default is False.
TYPE:
|
|
Filter for log messages. Default is None.
TYPE:
|
|
Whether to scrub cloud credentials from log records. Default is True.
TYPE:
|
|
Whether to enable context propagation via
:func:
TYPE:
|
|
Maximum size of the async logging queue when
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
log_level |
Current logging level.
TYPE:
|
enable_json |
Whether JSON formatting is enabled.
TYPE:
|
limiter |
Rate limiter instance (if rate limiting is enabled).
TYPE:
|
metrics_handler |
Metrics handler (if metrics are enabled).
TYPE:
|
log_directory |
Directory where log files are stored.
TYPE:
|
log_file |
Name of the log file.
TYPE:
|
log_file_path |
Full path to the log file.
TYPE:
|
Examples:
>>> from pylogshield import get_logger
>>> logger = get_logger("my_app", log_level="INFO", enable_json=True)
>>> logger.info({"user": "john", "password": "secret123"}, mask=True)
# Output: {"user": "john", "password": "***"}
| METHOD | DESCRIPTION |
|---|---|
info |
|
debug |
|
warning |
|
error |
|
critical |
|
exception |
Log an error with exception info. |
set_log_level |
Change the logger and handler levels at runtime. |
get_metrics |
Return logging metrics if metrics are enabled. |
context |
Return a sync context manager that injects fields into all logs. |
async_context |
Return an async context manager that injects fields into all logs. |
shutdown |
Stop any background listener and clean up resources. |
from_config |
Create a PyLogShield instance from a dictionary configuration. |
add_sensitive_fields |
Add field names to the sensitive data redaction registry. |
Source code in src/pylogshield/core.py
| Python | |
|---|---|
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 | |
info(msg: Any, *args: Any, mask: bool = False, **kwargs: Any) -> None
¶
debug(msg: Any, *args: Any, mask: bool = False, **kwargs: Any) -> None
¶
warning(msg: Any, *args: Any, mask: bool = False, **kwargs: Any) -> None
¶
error(msg: Any, *args: Any, mask: bool = False, **kwargs: Any) -> None
¶
critical(msg: Any, *args: Any, mask: bool = False, **kwargs: Any) -> None
¶
exception(msg: Any, *args: Any, mask: bool = False, exc_info: bool = True, **kwargs: Any) -> None
¶
Log an error with exception info.
When mask=True, string args of the active exception are scrubbed
before the record is emitted.
Note: traceback locals are not masked — use a traceback-filtering tool if local variable values must also be redacted.
Source code in src/pylogshield/core.py
set_log_level(level: Union[LogLevel, str, int]) -> None
¶
Change the logger and handler levels at runtime.
| PARAMETER | DESCRIPTION |
|---|---|
|
The new logging level to set.
TYPE:
|
Source code in src/pylogshield/core.py
get_metrics() -> Optional[Dict[str, Any]]
¶
Return logging metrics if metrics are enabled.
| RETURNS | DESCRIPTION |
|---|---|
dict or None
|
Dictionary with log counts and rates per level, total count, and elapsed time. Returns None if metrics are disabled. |
Examples:
>>> logger = get_logger("app", enable_metrics=True)
>>> logger.info("test")
>>> metrics = logger.get_metrics()
>>> print(f"Total logs: {metrics['count']}")
Source code in src/pylogshield/core.py
context(**fields: Any)
¶
Return a sync context manager that injects fields into all logs.
Shorthand for :func:~pylogshield.context.log_context. Requires the
logger to have been created with enable_context=True.
| PARAMETER | DESCRIPTION |
|---|---|
|
Key/value pairs to attach to every log record emitted inside the
TYPE:
|
Examples:
>>> with logger.context(request_id="abc", user_id=42):
... logger.info("Processing order")
Source code in src/pylogshield/core.py
async_context(**fields: Any)
¶
Return an async context manager that injects fields into all logs.
Shorthand for :func:~pylogshield.context.async_log_context.
Requires the logger to have been created with enable_context=True.
| PARAMETER | DESCRIPTION |
|---|---|
|
Key/value pairs to attach to every log record emitted inside the
TYPE:
|
Examples:
>>> async with logger.async_context(request_id="xyz"):
... logger.info("Async handler")
Source code in src/pylogshield/core.py
shutdown() -> None
¶
Stop any background listener and clean up resources.
This should be called when the logger is no longer needed to properly stop the background queue listener thread if async logging is enabled.
Source code in src/pylogshield/core.py
from_config(name: str, config: Mapping[str, Any]) -> 'PyLogShield'
classmethod
¶
Create a PyLogShield instance from a dictionary configuration.
| PARAMETER | DESCRIPTION |
|---|---|
|
The name of the logger.
TYPE:
|
|
Configuration dictionary with optional keys: level, enable_json, use_queue, use_rich, rate_limit_seconds, log_directory, log_file, rotate_file, rotate_max_bytes, rotate_backup_count, add_console, enable_metrics, log_filter, enable_context_scrubber, enable_context, queue_maxsize.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
PyLogShield
|
A new PyLogShield instance configured from the provided dictionary. |
Examples:
>>> config = {"level": "DEBUG", "enable_json": True, "rotate_file": True}
>>> logger = PyLogShield.from_config("my_app", config)
Source code in src/pylogshield/core.py
add_sensitive_fields(fields: List[str]) -> None
staticmethod
¶
Add field names to the sensitive data redaction registry.
| PARAMETER | DESCRIPTION |
|---|---|
|
List of field names to add to the sensitive registry.
TYPE:
|
Examples:
Source code in src/pylogshield/core.py
get_logger Function¶
get_logger(name: str = 'default_logger', *, force: bool = False, **kwargs: Any) -> PyLogShield
¶
Return a named PyLogShield instance, creating it if necessary.
This is the recommended way to obtain a PyLogShield instance. It integrates with Python's logging manager to ensure logger names are unique and reusable.
| PARAMETER | DESCRIPTION |
|---|---|
|
Logger name. Default is "default_logger".
TYPE:
|
|
If True, replace a non-PyLogShield with the same name. Default is False.
TYPE:
|
|
Additional arguments passed to the PyLogShield constructor when creating a new instance (e.g., log_level, enable_json, use_rich).
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
PyLogShield
|
A PyLogShield instance with the specified name. |
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If a logger with the name exists but is not a PyLogShield and force=False. |
Examples:
>>> logger = get_logger("my_app", log_level="DEBUG", enable_json=True)
>>> logger.info("Application started")
>>> # Get the same logger instance later
>>> same_logger = get_logger("my_app")
>>> same_logger is logger
True