python-best-practices

Python Community Python >=3.11 Best Practices

Comprehensive performance optimization guide for Python >=3.11 applications. Contains 45 rules across 9 categories, prioritized by impact to guide automated refactoring and code generation.

When to Apply

Reference these guidelines when:

• Writing new Python async/await code
• Processing large datasets or files
• Implementing caching and memoization
• Choosing data structures for performance
• Reviewing code for performance issues

Rule Categories by Priority

Quick Reference

1. I/O Patterns (CRITICAL)

• io-async-file-operations - Use async file I/O for non-blocking operations
• io-batch-database-operations - Batch database operations to reduce round trips
• io-connection-pooling - Use connection pooling for database and HTTP clients
• io-streaming-large-files - Stream large files instead of loading into memory
• io-use-buffered-io - Use buffered I/O for frequent small writes

2. Async Concurrency (CRITICAL)

• async-avoid-async-overhead - Avoid async overhead for CPU-bound work
• async-avoid-blocking-calls - Avoid blocking calls in async code
• async-create-task-fire-forget - Store references to fire-and-forget tasks
• async-gather-independent-operations - Use asyncio.gather() for independent operations
• async-semaphore-rate-limiting - Use semaphores for concurrency limiting
• async-taskgroup-structured-concurrency - Use TaskGroup for structured concurrency

3. Memory Management (HIGH)

• mem-avoid-intermediate-lists - Avoid intermediate lists in pipelines
• mem-generators-lazy-evaluation - Use generators for lazy evaluation
• mem-preallocate-lists - Preallocate lists when size is known
• mem-slots-dataclass - Use slots for memory-efficient classes
• mem-string-interning - Leverage string interning for repeated strings
• mem-weak-references - Use weak references for caches and observers

4. Data Structures (HIGH)

• ds-counter-for-counting - Use Counter for frequency counting
• ds-deque-for-queues - Use deque for O(1) queue operations
• ds-dict-get-default - Use dict.get() with default instead of KeyError handling
• ds-namedtuple-immutable-records - Use NamedTuple for immutable lightweight records
• ds-set-for-membership - Use Set for O(1) membership testing

5. Algorithm Efficiency (MEDIUM-HIGH)

• algo-avoid-repeated-computation - Cache expensive computations in loops
• algo-builtin-functions - Use built-in functions over manual implementation
• algo-itertools-recipes - Use itertools for efficient iteration patterns
• algo-list-comprehension - Use list comprehensions over manual loops
• algo-local-variable-lookup - Use local variables in hot loops
• algo-string-join - Use str.join() for string concatenation

6. Concurrency Model (MEDIUM)

• conc-asyncio-queues - Use asyncio.Queue for producer-consumer patterns
• conc-avoid-lock-contention - Minimize lock contention in threaded code
• conc-choose-right-model - Choose the right concurrency model
• conc-process-pool-chunking - Use chunking for ProcessPoolExecutor
• conc-thread-safe-globals - Use thread-safe data structures for shared state

7. Serialization (MEDIUM)

• serial-avoid-pickle-security - Avoid pickle for untrusted data
• serial-msgpack-binary - Use MessagePack for compact binary serialization
• serial-orjson-over-json - Use orjson for high-performance JSON
• serial-pydantic-validation - Use Pydantic for validated deserialization

8. Caching and Memoization (LOW-MEDIUM)

• cache-avoid-over-caching - Avoid over-caching low-value operations
• cache-cached-property - Use cached_property for expensive computed attributes
• cache-lru-cache-decorator - Use lru_cache for expensive pure functions
• cache-ttl-expiration - Implement TTL for time-sensitive caches

9. Runtime Tuning (LOW)

• runtime-avoid-global-lookups - Avoid repeated global and module lookups
• runtime-exception-handling-cost - Minimize exception handling in hot paths
• runtime-profile-before-optimizing - Profile before optimizing
• runtime-use-python311-plus - Upgrade to Python 3.11+ for free performance

How to Use

Read individual reference files for detailed explanations and code examples:

• Section definitions - Category structure and impact levels
• Rule template - Template for adding new rules

Each rule file contains:

• Brief explanation of why it matters
• Incorrect code example with explanation
• Correct code example with explanation
• Additional context and references

Reference Files