Whether you're looking to boost your Python application's speed, ensure memory safety, or simply be curious about integrating Rust into your workflow, this blog is your first step towards mastering the art of leveraging Rust's capabilities within your Python projects. Dive in now to discover a step-by-step guide that demystifies the process, making it accessible and straightforward for developers of all skill levels. Don't miss out on unlocking the full potential of your applications; join us in this exciting adventure!

Creating a dynamic library in Rust and calling it from Python using Foreign Function Interface (FFI) is an exciting way to leverage Rust's performance and safety in Python applications. This tutorial will guide you through the process step-by-step, with clear examples and explanations tailored for developers.

Creating a Rust Dynamic Library

Step 1: Setting Up Your Rust Project

First, you'll need to create a new Rust library project. Open a terminal and run the following command:

cargo new --lib rust_library
cd rust_library

This command creates a new Rust library project called rust_library and moves it into the project directory.

Step 2: Writing Your Rust Function

Open the src/lib.rs file and replace its contents with the following code, which defines a simple function to add two numbers:

#[no_mangle]
pub extern "C" fn add_numbers(a: i32, b: i32) -> i32 {
    a + b
}

• #[no_mangle] prevents Rust from changing the name of the function during compilation, making it easier to link from Python.

• pub extern "C" makes this function adhere to the C calling convention, a requirement for FFI.

Step 3: Configuring Cargo.toml

Open the Cargo.toml file and add the following lines to create a dynamic library:

[lib]
crate-type = ["cdylib"]

This configuration tells Rust to compile the library as a C-compatible dynamic library (cdylib).

Step 4: Building the Dynamic Library

Run the following command in your terminal:

cargo build --release

After the build completes, you'll find the dynamic library in target/release (e.g., librust_library.so on Linux, rust_library.dll on Windows, or librust_library.dylib on macOS).

Calling Rust from Python

Step 1: Installing ctypes in Python

Ensure you have ctypes available in your Python environment. It's included in the standard library, so you should be all set.

Step 2: Writing Python Code to Call Rust Function

Create a new Python file (e.g., call_rust.py) and write the following code:

from ctypes import cdll, c_int

# Load the shared library. Adjust the path as necessary.
lib = cdll.LoadLibrary("target/release/librust_library.so")

# Set the argument and return types
lib.add_numbers.argtypes = (c_int, c_int)
lib.add_numbers.restype = c_int

# Call the Rust function
result = lib.add_numbers(5, 7)
print(f"The result is: {result}")

• Adjust the path to LoadLibrary based on your dynamic library's location and name.

• argtypes and restype ensure Python correctly converts types between Python and Rust.

Step 3: Running Your Python Code

Run your Python script:

python call_rust.py

You should see the output:

The result is: 12

Conclusion

You've now successfully created a Rust dynamic library and called it from Python using FFI! This approach enables you to combine Rust's performance and reliability with Python's ease of use, creating powerful and efficient applications.

This tutorial serves as a basic introduction to FFI and inter-language communication. As you dive deeper, you'll encounter more complex scenarios, such as passing strings or complex data structures between Rust and Python. Each step has been designed to be as developer-friendly as possible, providing you with a solid foundation for integrating Rust and Python in your projects.

The examples for the above tutorial is referenced from : https://github.com/theArjun/learning-rust/tree/main/rust_adder

1. Dataclasses

Syntax:

from dataclasses import dataclass

@dataclass
class Point:
    x: float
    y: float
    z: float = 0.0

Key Features:

• Concise syntax with the @dataclass decorator.

• Automatically generates special methods (e.g., __init__, __repr__) based on class attributes.

• Default values for attributes can be specified.

Use Cases:

• Simple data structures where automatic methods generation is sufficient.

2. attrs

Syntax:

import attr

@attr.s
class Point:
    x = attr.ib()
    y = attr.ib()
    z = attr.ib(default=0.0)

Key Features:

• Decorator @attr.s is used to define a class with attributes.

• Explicit attribute definition using attr.ib() with optional default values.

• Powerful features like validation, converters, and metadata.

Use Cases:

• Fine-grained control over attribute behavior.

• Rich attribute features like validation and metadata.

3. Pydantic

Syntax:

from pydantic import BaseModel

class Point(BaseModel):
    x: float
    y: float
    z: float = 0.0

Key Features:

• Inherited from BaseModel.

• Automatic validation based on type annotations.

• Supports parsing and serialization of data from/to various formats (e.g., JSON).

Use Cases:

• Data validation and parsing in applications involving input from external sources.

• API request/response handling where automatic parsing and validation are essential.

Comparison

1. Syntax and Ease of Use

• Dataclasses: Simple and concise syntax with the @dataclass decorator.

• attrs: Explicit attribute definition using attr.ib(), providing fine-grained control.

• Pydantic: Concise syntax with automatic validation based on type annotations.

2. Features

• Dataclasses: Automatic generation of special methods; default values can be specified.

• attrs: Rich set of features, including validation, converters, and metadata.

• Pydantic: Automatic validation, parsing, and serialization; supports JSON schema generation.

3. Performance

• Dataclasses: Generally lightweight and performs well.

• attrs: Slightly heavier due to additional features, but still performs well.

• Pydantic: May have higher overhead due to additional functionality; suitable for scenarios where validation and parsing are crucial.

4. Use Cases

• Dataclasses: Simple data structures where automatic methods generation suffices.

• attrs: Situations requiring fine-grained control over attribute behavior and additional features.

• Pydantic: Data validation, parsing, and serialization in scenarios involving external data sources, APIs, or configuration files.

Conclusion

Choosing between dataclasses, attrs, and pydantic depends on the specific needs of the project. For basic data structures, dataclasses provide a lightweight and easy-to-use solution. attrs is a good choice when fine-grained control over attributes and additional features are required. On the other hand, pydantic is well-suited for scenarios involving data validation, parsing, and serialization, especially in applications dealing with external data sources or APIs. Ultimately, the choice depends on the trade-offs between simplicity, features, and performance based on the project requirements.

Setting Up Your Environment

Before we begin, ensure you have Python installed. FastAPI runs on Python 3.6+.

1. Create a Virtual Environment (Optional but Recommended): This keeps your project dependencies separate from your global Python installation.

    python -m venv venv
    source venv/bin/activate  # On Windows use `venv\Scripts\activate`
   

2. Install FastAPI and Uvicorn: Uvicorn is an ASGI server that will run our FastAPI application.

    pip install fastapi uvicorn
   

3. Install Additional Dependencies: We'll use aiofiles for asynchronous file handling.

    pip install aiofiles
   

Building the FastAPI Backend

The FastAPI App

We will create a file namedmain.py and write the FastAPI application code such that it sends new content from the Apache logs when it's added. We also need to implement a way to watch the file for changes and send those changes over the WebSocket. This can be achieved using file polling or more sophisticated file monitoring techniques. However, keep in mind that Python doesn't have built-in, cross-platform support for real-time file monitoring, so we'll use a simple polling approach for this example.

FastAPI Backend with File Polling

Here's how you can adjust the main.py to include file polling:

from fastapi import FastAPI, WebSocket
from fastapi.responses import HTMLResponse
import aiofiles
import asyncio
import os

app = FastAPI()

html = """
<!DOCTYPE html>
<html>
    <head>
        <title>Apache Logs Stream</title>
    </head>
    <body>
        <h1>Apache Access Log</h1>
        <pre id="accessLog"></pre>
        <script src="/static/app.js"></script>
    </body>
</html>
"""

@app.get("/")
async def get():
    return HTMLResponse(html)

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    file_path = '/path/to/apache.access.log'
    last_size = 0

    while True:
        await asyncio.sleep(1)  # Poll every second
        current_size = os.path.getsize(file_path)
        if current_size > last_size:
            async with aiofiles.open(file_path, mode='r') as f:
                await f.seek(last_size)
                while new_line := await f.readline():
                    # If new line is just a blank line, new line char or empty string, skip it
                    if new_line in ["\n", "\r\n", ""]:
                        continue
                    await websocket.send_text(new_line)
            last_size = current_size

In this code:

1. The WebSocket connection is established.

2. It enters an infinite loop where it checks the file size every second.

3. If the file size has increased (indicating new content), it reads the new lines and sends them over the WebSocket.

Remember to replace /path/to/apache.access.log with the actual path to your Apache access log file.

Running the Application

Run your FastAPI server as before:

uvicorn main:app --reload

Now, when you add new lines to your Apache log file, those lines will be sent to the connected web client in real-time.

Important Considerations

• Polling vs. Event-Driven Monitoring: This example uses polling, which is less efficient than event-driven monitoring. For a production environment, you might want to explore libraries like watchdog for Python, which can provide more efficient file monitoring.

• Error Handling: In a production environment, add error handling to deal with issues like lost WebSocket connections or file read errors.

• Security and Performance: Be cautious about security implications and the performance impact of constantly reading large log files, especially in a busy production environment.Replace /path/to/apache.access.log with the path to your Apache log file.

The Frontend JavaScript

In a folder named static, create app.js with the following content:

const ws = new WebSocket('ws://localhost:8000/ws');
ws.onmessage = function(event) {
    document.getElementById('accessLog').textContent += event.data + '\n';
};

Key Takeaways

• Asynchronous Streaming: FastAPI and aiofiles make it easy to handle real-time data streaming asynchronously.

• Security: Remember to implement authentication and secure your WebSocket connections, especially in a production environment.

• Performance: FastAPI is known for its high performance, making it an excellent choice for real-time data streaming applications.

Conclusion

Using FastAPI for streaming Apache logs to a web interface is a powerful approach. It's not only efficient but also provides a modern Pythonic way of handling real-time data. This setup can be customized to fit a variety of monitoring and streaming needs.

Happy coding and real-time monitoring with FastAPI! 🐍💨

The Quest Begins: The Simple Yet Tricky Date Function

Let's set the stage with our star player, a simple function named get_today_date. Its mission is straightforward – to fetch and return the current date. Here's how it looks in Python:

from datetime import datetime

def get_today_date():
    return datetime.now().date()

It's a simple piece of code, but as we'll see, even the simplest code can lead to complex testing scenarios.

The Twist: Testing a Moving Target

Testing static functions is a breeze, but testing a function that returns a dynamic value like the current date? That's where the real challenge lies. We need our tests to be consistent, regardless of the actual date. This is where I introduce a clever trick up my Python sleeve – unittest.mock.

The Magic Trick: Mocking the Date

Using the unittest framework, along with unittest.mock, I conjure a controlled environment where datetime.now() is under our spell, always returning a fixed, predictable date.

Here's a test case for our get_today_date function:

import unittest
from unittest.mock import patch
from datetime import datetime
from my_module import get_today_date  # Replace 'my_module' with your actual module name

class TestGetTodayDate(unittest.TestCase):
    @patch('my_module.datetime')  # Again, replace 'my_module' with your actual module name
    def test_get_today_date(self, mock_datetime):
        mock_date = datetime(2024, 1, 19)
        mock_datetime.now.return_value = mock_date

        result = get_today_date()

        self.assertEqual(result, mock_date.date())

if __name__ == '__main__':
    unittest.main()

This test ensures our function is behaving as expected, but as with any magic, there are caveats.

The Caveats: Unveiling the Shortcomings

1. Over-Reliance on Mocking

By using unittest.mock, we're testing in an artificial bubble. It's perfect for consistency but can potentially hide issues in handling real-world date scenarios.

2. Ignoring Real-World Scenarios

The test blissfully skips over the ever-changing nature of real dates. It doesn't account for complexities like time zones or daylight saving time.

3. Maintenance Overhead

Setting up these mocks across multiple tests can be cumbersome, adding extra maintenance and complexity.

4. Risk of False Positives

Just because the test passes with the mocked date doesn't guarantee the absence of bugs in date handling.

5. Framework Limitations

This approach is tailored for unittest. Switching frameworks could mean a significant overhaul of our testing strategy.

The Conclusion: A Mixed Bag of Tricks

Despite these challenges, the mocking strategy is a valuable tool in our Python testing arsenal, especially for functions dealing with dates. However, it's crucial to complement this approach with additional testing strategies to ensure robustness and real-world applicability.

So there you have it – a glimpse into the intricate dance of testing date functions in Python. It's a blend of precision, foresight, and a bit of magic, all wrapped up in the pursuit of reliable, bug-resistant code. Keep experimenting, keep learning, and most importantly, keep enjoying the journey of coding! 🎩✨🐍💻📅

1. Create a maintenance middleware:

   • This middleware will intercept requests and redirect them to a maintenance page if a maintenance mode is active.

2. Add a Maintenance View and Template:

   • Create a simple view and template for the maintenance page that users will see.

3. Configure the middleware:

   • Add the middleware to your Django project settings.

4. Activate/Deactivate Maintenance Mode:

   • We can activate maintenance mode by changing a variable, a file existence, or a database flag.

Here's a simple example to illustrate these steps:

Step 1: Create a Maintenance Middleware

We will create a file namedmaintainence_middleware.py in one of your apps:

from django.http import HttpResponse
from django.shortcuts import redirect

class MaintenanceMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        # Check if maintenance mode is active
        # This can be a setting in settings.py, a database flag, or checking for a file
        if hasattr(settings, 'MAINTENANCE_MODE') and settings.MAINTENANCE_MODE:
            # Allow access to certain IPs
            allowed_ips = ['123.456.78.90',]  # Your IP here
            if request.META.get('REMOTE_ADDR') in allowed_ips:
                return self.get_response(request)
            else:
                return redirect('maintenance_page')
        return self.get_response(request)

Step 2: Add a Maintenance View and Template

In our views.py:

from django.shortcuts import render

def maintenance_page(request):
    return render(request, 'maintenance.html')

Create a maintenance.html template in your templates directory with the appropriate HTML content.

Step 3: Configure the Middleware

In your settings.py, add the middleware to MIDDLEWARE:

MIDDLEWARE = [
    # ...
    'yourapp.maintenance_middleware.MaintenanceMiddleware',
    # ...
]

Step 4: Activate/Deactivate Maintenance Mode

In your settings.py:

# Set this to True to activate maintenance mode
MAINTENANCE_MODE = False

Step 5: You might want to:

• Custom Maintenance Page: Customize the maintenance.html to inform our users about the maintenance and possibly provide an estimated time for when the service will be back.

• Dynamic Activation: Instead of manually changing the settings.py, we could use an environment variable or a database setting that we can modify without deploying new code.

• Logging Access Attempts: We might want to log attempts to access the site during maintenance for security and analysis.

• Testing: Ensure we test this thoroughly in a staging environment before deploying it to production, especially the IP whitelisting part.

• Notify Users in Advance: If possible, notify our users in advance about the planned maintenance.

By following these steps, you can implement a maintenance mode in your Django application that can be easily activated or deactivated as needed.

Step 6: Testing Maintainence Middleware

1. Set up the test environment: This involves creating a test class that inherits from TestCase and setting up any necessary configurations.

2. Test the middleware's behavior: You should test for both when maintenance mode is active and when it is not. Additionally, testing for access from allowed IPs and disallowed IPs is important.

Here's an example of how you might write these test cases:

1. Set Up the Test Environment

First, create a test file in your Django app's tests directory, e.g., test_middleware.py.

from django.test import TestCase, RequestFactory
from django.conf import settings
from yourapp.views import maintenance_page
from yourapp.maintenance_middleware import MaintenanceMiddleware

class MaintenanceMiddlewareTest(TestCase):

    def setUp(self):
        # The RequestFactory instance is a way to create request instances for use in testing.
        self.factory = RequestFactory()
        self.middleware = MaintenanceMiddleware(lambda x: x)

2. Test the Middleware's Behavior

Test When Maintenance Mode is Active

def test_maintenance_mode_active(self):
    # Set maintenance mode to active
    settings.MAINTENANCE_MODE = True

    # Create a request instance
    request = self.factory.get('/any-page')
    request.META['REMOTE_ADDR'] = '111.222.33.44'  # Non-allowed IP

    # Process the request through the middleware
    response = self.middleware(request)

    # Check that the response is a redirect to the maintenance page
    self.assertEqual(response.status_code, 302)
    self.assertEqual(response.url, '/maintenance-page')  # URL of your maintenance page

Test When Maintenance Mode is Inactive

def test_maintenance_mode_inactive(self):
    # Set maintenance mode to inactive
    settings.MAINTENANCE_MODE = False

    # Create a request instance
    request = self.factory.get('/any-page')

    # Process the request through the middleware
    response = self.middleware(request)

    # Check that the request passes through the middleware unaffected
    self.assertEqual(response.status_code, 200)

Test Access from Allowed IPs

def test_access_from_allowed_ip(self):
    # Set maintenance mode to active
    settings.MAINTENANCE_MODE = True

    # Create a request instance with an allowed IP
    request = self.factory.get('/any-page')
    request.META['REMOTE_ADDR'] = '123.456.78.90'  # Allowed IP

    # Process the request through the middleware
    response = self.middleware(request)

    # Check that the request passes through the middleware unaffected
    self.assertEqual(response.status_code, 200)

Additional Notes:

• Make sure to replace 'yourapp' with the actual name of your Django app.

• Modify the '/maintenance-page' with the actual URL of your maintenance page.

• These tests assume that the normal response (when not in maintenance mode) has a status code of 200. Adjust according to your actual app's behavior.

• Remember to reset any settings (like MAINTENANCE_MODE) after the test to avoid side effects on other tests.

By covering these scenarios, your test case will effectively validate the behavior of your maintenance middleware under different conditions.

How Middleware Works:

1. Layered Structure: Middleware in Django is like a series of layers or wrappers around the view. When a request comes in, it goes through each middleware in order before finally reaching the view. After the view has processed the request, the response travels back through the middleware stack to the user.

2. Request and Response Processing: Middleware can be used for preprocessing requests before they reach the view and for post-processing responses before they are returned to the client.

3. Global Effect: Unlike decorators, which are applied to individual views, middleware applies to all requests and responses, making them useful for tasks that need to be handled globally.

Common Uses of Middleware:

• Session Management: Handles the sending and receiving of cookies, which are used to manage sessions.

• Authentication: Associates users with requests using sessions.

• Cross-Site Request Forgery Protection: Provides a security mechanism to ensure that forms are submitted from authenticated users and from the website itself.

• Caching: Manages caching, which can store views or parts of views to improve performance.

• Localization: Detects user’s language and timezone preferences and adjusts content accordingly.

Writing Custom Middleware:

Creating custom middleware in Django involves defining a class with specific methods that Django recognizes:

• __init__(self, get_response): Constructor method called once when the web server starts.

• __call__(self, request): Called on each request, before Django calls any view.

• process_view(self, request, view_func, view_args, view_kwargs): Called just before Django calls the view.

• process_exception(self, request, exception): Called if an exception is raised by

the view.

• process_template_response(self, request, response): Called if the response instance has a render() method, indicating it's a TemplateResponse or equivalent.

Example of a simple custom middleware:

class MyCustomMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        # Code executed on each request before the view is called
        response = self.get_response(request)
        # Code executed on each request after the view is called
        return response

To use this middleware, you would add it to the MIDDLEWARE setting in your Django settings file.

Middleware and Django’s Design Philosophy:

Middleware in Django aligns with its design philosophy of being explicit, reusable, and pluggable. It allows developers to cleanly and efficiently modify the input or output of an application without cluttering the business logic in views or models. This makes the application easier to maintain and extend.

Creating Custom Middlewares

Creating detailed code for each custom middleware, along with its pros and cons and particular use cases, would be quite extensive. However, I can provide a high-level overview of a few examples, covering the basic implementation, pros and cons, and use cases for each.

1. Request and Response Logging Middleware

Create a middleware that logs the details of each request and response. This can include the request path, method, duration of the request, status code of the response, and IP address of the client. This is useful for debugging and monitoring the health of your application.

Code Overview:

import time
from django.utils.deprecation import MiddlewareMixin

class LoggingMiddleware(MiddlewareMixin):
    def process_request(self, request):
        request.start_time = time.time()

    def process_response(self, request, response):
        duration = time.time() - request.start_time
        print(f'Request to {request.path} took {duration} seconds')
        return response

Pros:

• Easy to implement and integrate.

• It helps in monitoring and debugging.

Cons:

• Increases log size.

• It might slow down request handling if logging is extensive.

Use Cases:

• Debugging performance issues.

• Monitoring API usage and response times.

2. API Throttling/Limiting Middleware

Implement rate limiting to restrict the number of requests a user can make to your API within a given timeframe. This helps to prevent abuse and ensures that your API remains responsive under high load.

Code Overview:

from django.http import JsonResponse
from django.core.cache import cache

class RateLimitMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        ip = request.META.get('REMOTE_ADDR')
        if cache.get(ip):
            return JsonResponse({'error': 'rate limit exceeded'}, status=429)
        else:
            cache.set(ip, 'exists', timeout=60)  # 1 request per minute
            response = self.get_response(request)
            return response

Pros:

• Prevents abuse of the API.

• Helps in managing server load.

Cons:

• Might block legitimate users if not configured properly.

• Additional overhead for maintaining the rate-limiting logic.

Use Cases:

• Public APIs where rate limiting is essential.

• Preventing DoS attacks.

3. Maintenance Mode Middleware

Develop middleware that enables a “maintenance mode” for your website. When this mode is active, the middleware can return a predefined maintenance response for all HTTP requests, except those from authenticated admins or from certain IP addresses.

Code Overview:

from django.http import HttpResponse
from django.conf import settings

class MaintenanceModeMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if settings.MAINTENANCE_MODE:
            return HttpResponse("Site under maintenance", status=503)
        response = self.get_response(request)
        return response

Pros:

• It is easy to switch the entire site to maintenance mode.

• A customizable response for maintenance.

Cons:

• Might accidentally block access to the site if not managed correctly.

• Requires additional settings and configurations.

Use Cases:

• Scheduled downtime for updates or maintenance.

• Emergency response to critical issues.

4. SSL/TLS Redirection Middleware

Ensure security by automatically redirecting HTTP requests to HTTPS, helping to keep user data secure during transit.

Code Overview:

from django.http import HttpResponsePermanentRedirect

class SSLRedirectMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if not request.is_secure():
            return HttpResponsePermanentRedirect(f'https://{request.get_host()}{request.get_full_path()}')
        response = self.get_response(request)
        return response

Pros:

• Ensures secure data transmission.

• It is easy to enforce HTTPS across the site.

Cons:

• Can lead to redirect loops if not configured correctly.

• Requires SSL/TLS certificate installation.

Use Cases:

• Websites that handle sensitive data (e-commerce, online banking).

• General enhancement of website security.

5. CORS Middleware for Specific Routes

While Django provides built-in support for managing Cross-Origin Resource Sharing (CORS), you might want a middleware that applies custom CORS policies to certain routes or APIs in your application.

Code Overview:

class CORSMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        if request.path.startswith('/api/'):  # Apply CORS only to API routes
            response["Access-Control-Allow-Origin"] = "*"
        return response

Pros:

• Allows control over cross-origin requests for specific routes.

• Enhances security by limiting CORS only where necessary.

Cons:

• Potential security risk if misconfigured.

• You might need to handle preflight requests separately.

Use Cases:

• Public APIs are meant to be accessed from different domains.

• Single-page applications (SPAs) that need to interact with APIs hosted on different domains.

6. Profile Middleware for Performance Monitoring

A middleware that measures the execution time of views and other parts of your request/response cycle. This can be invaluable for identifying performance bottlenecks.

Code Overview:

import time

class ProfileMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        start_time = time.time()
        response = self.get_response(request)
        end_time = time.time()
        print(f"Request to {request.path} took {end_time - start_time} seconds")
        return response

Pros:

• Helps in identifying slow parts of the application.

• Easy to integrate and remove as needed.

Cons:

• Adds extra processing time to each request.

• Not a detailed profiling tool; more suitable for high-level monitoring.

Use Cases:

• Identifying performance bottlenecks.

• Monitoring response times during load testing.

7. User-Agent Restriction Middleware

Create middleware that blocks or allows requests based on the user-agent string. This can be used to prevent certain web crawlers from accessing your site or to provide different responses for mobile and desktop users.

Detailed Code:

from django.http import HttpResponseForbidden

class UserAgentRestrictionMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        user_agent = request.META.get('HTTP_USER_AGENT', '')
        if 'DisallowedUserAgent' in user_agent:
            return HttpResponseForbidden("Access Denied")
        response = self.get_response(request)
        return response

Pros:

• Control access based on user-agent.

• Block unwanted crawlers or bots.

Cons:

• User agents can be spoofed.

• Might accidentally block legitimate users.

Use Cases:

• Blocking scraping bots.

• Providing different content for mobile and desktop users.

8. Localization and Timezone Middleware

Enhance the user experience by automatically determining the user’s preferred language and timezone based on their browser settings or location. Use this information to localize content and display times in the user’s local time zone.

Detailed Code:

from django.utils import timezone
from django.utils.translation import activate

class LocalizationMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        # Set language
        user_language = request.META.get('HTTP_ACCEPT_LANGUAGE')
        activate(user_language)

        # Set timezone
        user_timezone = request.COOKIES.get('timezone')
        if user_timezone:
            timezone.activate(user_timezone)
        else:
            timezone.deactivate()

        response = self.get_response(request)
        return response

Pros:

• Enhances the user experience with language and timezone personalization.

• Improves the usability of global applications.

Cons:

• Requires additional logic for timezone detection.

• Potential overhead in detecting and setting preferences.

Use Cases:

• Multi-language websites.

• Applications requiring content based on the user's locale.

9. Content Compression Middleware

Implement middleware to compress the response data before sending it to the client. This can help reduce bandwidth usage and improve load times, especially for users with slower internet connections.

Detailed Code:

import gzip
from io import BytesIO

class GZipMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)

        if 'gzip' not in request.META.get('HTTP_ACCEPT_ENCODING', ''):
            return response

        if response.status_code != 200 or len(response.content) < 200:
            return response

        buffer = BytesIO()
        gzip_file = gzip.GzipFile(mode='wb', fileobj=buffer)
        gzip_file.write(response.content)
        gzip_file.close()

        response.content = buffer.getvalue()
        response['Content-Encoding'] = 'gzip'
        response['Content-Length'] = str(len(response.content))

        return response

Pros:

• Reduces the size of the response, saving bandwidth.

• Improves loading times for clients with slow connections.

Cons:

• Adds processing overhead on the server.

• Not all clients may support gzip encoding.

Use Cases:

• Websites with high traffic and large response sizes.

• Improving performance for users with slower internet connections.

10. A/B Testing Middleware

Develop middleware for conducting A/B tests. You can use this to present different versions of your site to different users and gather data on how each version performs.

Detailed Code:

import random
from django.utils.deprecation import MiddlewareMixin

class ABTestingMiddleware(MiddlewareMixin):
    def process_request(self, request):
        # Assign a random version for A/B testing
        if 'ABTestVersion' not in request.session:
            request.session['ABTestVersion'] = random.choice(['A', 'B'])

        request.ab_test_version = request.session['ABTestVersion']

    def process_template_response(self, request, response):
        if hasattr(request, 'ab_test_version'):
            response.context_data['ABTestVersion'] = request.ab_test_version
        return response

Pros:

• Simple implementation of A/B testing.

• Ability to test different versions of your site simultaneously.

Cons:

• May require additional logic for complex testing scenarios.

• Tracking and analysis of results need to be handled separately.

Use Cases:

• Testing new features or design changes.

• Optimizing the user experience and conversion rates.

11. Authentication and Authorization Middleware

Create middleware to handle custom authentication and authorization beyond Django's built-in systems. For example, implementing token-based authentication or integrating with OAuth providers.

Detailed Code:

from django.http import JsonResponse
from django.contrib.auth import authenticate

class TokenAuthMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        token = request.META.get('HTTP_AUTHORIZATION')
        if token:
            user = authenticate(request, token=token)
            if user:
                request.user = user
            else:
                return JsonResponse({'error': 'Invalid token'}, status=401)
        response = self.get_response(request)
        return response

Pros:

• Customizable authentication process.

• Flexibility to integrate with different token systems.

Cons:

• Increased complexity in managing authentication.

• Responsibility to ensure the security of the custom implementation.

Use Cases:

• Integrating third-party authentication systems.

• Implementing token-based authentication for APIs.

Each middleware serves a distinct purpose and can significantly enhance the functionality of a Django application. The choice of middleware should be based on the specific requirements of the project and balanced against potential drawbacks, such as performance overhead or complexity.

Understanding Profiling in Django

Profiling is the process of measuring the various aspects of program performance, such as memory usage and execution time, to identify bottlenecks. In Django, several tools can be used for profiling.

Tools for Profiling:

• Django Debug Toolbar: This is a configurable set of panels that display various debug information about the current request/response.

    # Installation
    pip install django-debug-toolbar
  
    # settings.py
    INSTALLED_APPS = [
        # ...
        'debug_toolbar',
    ]
    MIDDLEWARE = [
        # ...
        'debug_toolbar.middleware.DebugToolbarMiddleware',
    ]
  

• cProfile Module: A built-in Python module that can profile any Python program.

    import cProfile
    cProfile.run('your_function()')
  

Steps for Profiling:

1. Identify Slow Areas: Use Django Debug Toolbar to get an overview of request times, SQL queries, and more.

2. Detailed Profiling: For a detailed analysis, use cProfile to profile specific functions or code blocks.

3. Analyze the Output: Look for functions or queries that take the most time or are called excessively.

Detecting and Optimizing Lagging Code

Once you've profiled your application, the next step is to optimize the lagging parts.

Common Issues and Solutions:

• Inefficient Database Queries: Use Django’s select_related and prefetch_related to optimize SQL queries.

    # Optimizing ForeignKey relationships
    queryset = MyModel.objects.select_related('foreign_key_field')
    # Optimizing ManyToMany fields
    queryset = MyModel.objects.prefetch_related('many_to_many_field')
  

• Reducing Query Counts: Aim to reduce the number of queries per request. Cache frequently used data.

    from django.core.cache import cache
  
    def my_view(request):
        data = cache.get('my_data')
        if not data:
            data = MyModel.objects.get_expensive_query()
            cache.set('my_data', data, timeout=300)  # Cache for 5 minutes
        return render(request, 'my_template.html', {'data': data})
  

• Optimizing Templates: Reduce template complexity, use template fragment caching.

    {% load cache %}
    {% cache 500 cache_fragment_name %}
     ... expensive calculations ... 
    {% endcache %}
  

Profiling Python Code

Using the cProfile module, you can get a detailed report of function calls and execution times. This helps in identifying bottlenecks in your Python code.

import cProfile
import io
import pstats

def profile_your_function():
  # Your function code here

# Create a Profile object and run your function
profiler = cProfile.Profile()
profiler.enable()
profile_your_function()
profiler.disable()

# Generate profiling report
s = io.StringIO()
ps = pstats.Stats(profiler, stream=s).sort_stats('cumulative')
ps.print_stats()
print(s.getvalue())

Introducing Asynchronous Task Queues

For tasks that are time-consuming or can be processed in the background, asynchronous task queues are a lifesaver. They help in offloading such tasks from the main thread, improving the responsiveness of your application.

Celery: A Robust Task Queue

Celery is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background.

Setting Up Celery with Django:

1. Install Celery:

    pip install celery
   

2. Configure Celery in your Django project:

   Create a new file celery.py in your Django project’s main directory.

    from __future__ import absolute_import, unicode_literals
    import os
    from celery import Celery
   
    # Set the default Django settings module for the 'celery' program.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')
   
    app = Celery('your_project')
   
    # Using a string here means the worker doesn't have to serialize
    # the configuration object to child processes.
    app.config_from_object('django.conf:settings', namespace='CELERY')
   
    # Load task modules from all registered Django app configs.
    app.autodiscover_tasks()
   
    @app.task(bind=True)
    def debug_task(self):
        print(f'Request: {self.request!r}')
   

3. Create a Task:

   In your Django app, create a tasks.py file and define your asynchronous tasks.

    from celery import shared_task
   
    @shared_task
    def add(x, y):
        return x + y
   

4. Run Celery Worker:

   Run the Celery worker process to listen for incoming task requests.

    celery -A your_project worker --loglevel=info
   

Using Celery in Views:

Call your asynchronous task from Django views or models. The task will be executed in the background by Celery workers.

from .tasks import add

def some_view(request):
    # Call an asynchronous task
    add.delay(4, 4)
    return HttpResponse("Task is running in the background")

Conclusion

Performance optimization in Django is a continuous process. By profiling your application regularly, optimizing the lagging parts, and using asynchronous task queues like Celery, you can significantly improve your Django application's performance. Remember, a fast and efficient application not only provides a better user experience but also contributes to higher scalability and maintainability.

Understanding Django ORM

Django's ORM allows developers to interact with the database using Python objects, abstracting the underlying SQL. However, this abstraction can lead to inefficient queries if not managed properly.

1. Query Efficiency

• Select Related and Prefetch Related: One common issue in Django ORM is the "N+1" query problem, which occurs when a loop executes a new query for each iteration. To mitigate this, use select_related for foreign key and one-to-one relationships, and prefetch_related for many-to-many and reverse foreign key relationships.

• Only and Defer: If you only need a subset of fields from the database, only() can be used to load specific fields, reducing the amount of data transferred. Conversely, defer() delays loading of specific fields until they are accessed.

• Aggregates and Annotations: Aggregation functions like Sum, Count, etc., can be used to perform calculations at the database level. Annotations allow for complex queries, like conditional aggregates, without pulling excessive data into Python.

• F() Expressions: Use F() expressions to perform database-side operations, reducing Python's processing load.

2. Indexing and Database Design

• Appropriate Indexing: Indexes are crucial for query optimization. Identify frequently queried fields, especially in WHERE, ORDER BY, and JOIN clauses, and index them accordingly.

• Database Normalization: While normalization reduces data redundancy, over-normalization can lead to excessive joins. Conversely, denormalization can improve read performance at the cost of write performance and data integrity. Strike a balance based on your application's read-write patterns.

3. Efficient Data Retrieval

• Paginate Results: For views that display large datasets, implement pagination to limit the number of records retrieved per request.

• Caching: Implement caching strategies for data that doesn't change often. This can be done at various levels - per-view, per-query, or even using low-level cache frameworks.

4. Understanding Query Execution

• Query Evaluation: Django's querysets are lazy, meaning they are not executed until evaluated. Understanding when a queryset is evaluated can help prevent unnecessary database hits.

• Use of exists(): If you only need to check if a queryset has results, exists() is more efficient than loading the entire queryset.

5. Asynchronous ORM Support (Django 3.1 and later)

• Async ORM: Django 3.1 introduced support for asynchronous ORM. This allows for asynchronous query execution, which can be beneficial for IO-bound operations.

6. Profiling and Debugging

• Query Logging: Use Django's logging framework to log queries in development. Tools like Django Debug Toolbar can provide insights into query patterns and inefficiencies.

• Profiling Tools: Use profiling tools to understand the performance characteristics of your ORM operations.

7. Best Practices in Model Design

• Lean Models: Keep models focused and lean. Avoid unnecessary fields and relationships.

• Using select_for_update(): In scenarios where transaction integrity is critical, such as in concurrent environments, select_for_update() can be used to lock rows until the transaction is complete.

8. Optimizing ORM for Scalability

• Database Sharding and Replication: For large-scale applications, consider database sharding or replication. Django ORM can be configured to work with multiple databases, allowing for scalability and improved performance.

• Batch Processing: For operations on large datasets, consider batch processing. The bulk_create, bulk_update, and iterator() methods can be used to efficiently handle large volumes of data.

9. Avoiding ORM Anti-Patterns

• Overusing all(): Avoid using all() to fetch entire tables, especially for large datasets.

• Raw SQL Queries: While Django ORM is powerful, sometimes raw SQL queries are necessary for complex operations. Use them judiciously.

10. Regular Performance Audits

• Regular Audits: Conduct regular performance audits on your ORM usage. This includes reviewing query patterns, indexing strategies, and overall database design.

In conclusion, optimizing Django ORM is a multifaceted process involving query efficiency, database design, data retrieval techniques, and an understanding of Django's underlying mechanisms.

1. Set Up Django Channels

Firstly, ensure that Django Channels is properly installed and configured in your Django project. This includes setting up the ASGI application and routing.

2. Create Test Cases

You will need to create test cases specifically for your Channels consumers. Django Channels offers a ChannelsLiveServerTestCase class that you can use to write tests for your consumers.

3. Establish a Test WebSocket Connection

Using a test client (like the one provided by Channels or a third-party client such as websocket-client), establish a WebSocket connection to your Channels server. This is typically done in the setup phase of your test case.

4. Implement User Authentication

For authenticating users, you can use Django's standard authentication mechanisms. When a WebSocket connection is made, the consumer can access the user from the scope. However, during testing, you might need to simulate this.

You can simulate a logged-in user in tests by creating a user and then manually setting the appropriate cookies or session data in the test client. This will depend on your authentication method (e.g., token-based, session-based).

5. Testing with Authenticated User

Once you have set up the user's authentication context, you can proceed to test your consumer's behavior with the authenticated user. Make sure to test for both authenticated and unauthenticated scenarios.

6. Clean Up

After your tests, ensure to clean up any created data or states to maintain test isolation.

Example Code Snippet

Here is a very basic example of how a test case might look:

from channels.testing import ChannelsLiveServerTestCase
from django.contrib.auth.models import User
from myapp.consumers import MyConsumer

class MyConsumerTestCase(ChannelsLiveServerTestCase):
    def setUp(self):
        # Set up user and authentication here
        self.user = User.objects.create_user(username='testuser', password='password')
        self.client.force_login(self.user)
        # More setup as needed

    def test_my_consumer(self):
        # Connect to the WebSocket
        with self.client.websocket_connect("/ws/myapp/") as websocket:
            # Test sending and receiving messages
            websocket.send_json({"type": "test_message"})
            response = websocket.receive_json()
            self.assertEqual(response, {"type": "response", "authenticated": True})
            # More assertions and tests

Important Notes

• Ensure that your test environment correctly mirrors your production environment, especially regarding the authentication setup.

• Depending on the complexity of your application, you might need additional setup for handling channels layers, database interactions, etc.

This is a basic guide, and the specifics can vary significantly based on the exact requirements and setup of your Django project.

1. Use Descriptive Names: Choose branch names that clearly convey the purpose of the branch. Use meaningful words or phrases to describe the changes or features being worked on.

2. Keep It Short: While descriptive, the branch names should also be concise. Long branch names can be cumbersome and may not display fully in certain tools.

3. Use Hyphens or Underscores: Use hyphens (-) or underscores (_) to separate words in branch names. Avoid using spaces or special characters that may cause issues with some systems.

4. Include a Prefix: Consider adding a prefix to your branch names to categorize them. For example:

   • feature/new-login-page

   • bugfix/issue-123

   • hotfix/critical-bug

5. Use Lowercase Letters: Stick to lowercase letters for branch names. Git is case-sensitive, and using consistent lowercase makes it easier to avoid conflicts.

6. Avoid Using Personal Names: Don't include personal names or usernames in branch names, as it doesn't provide any information about the branch's purpose.

7. Use Issue Tracker References: If your team uses an issue tracker (like JIRA, GitHub Issues, etc.), consider referencing the issue number in the branch name. This helps link branches to specific tasks or problems.

8. Follow a Workflow: If your team follows a specific Git workflow (e.g., Gitflow), adhere to the naming conventions recommended in that workflow.

9. Delete Merged Branches: Once a branch has been merged into the main branch (e.g., master or main), consider deleting it to keep the repository clean and reduce clutter.

Examples of good branch names:

• feature/user-authentication

• bugfix/issue-456

• refactor/update-db-models

Examples of poor branch names:

• branch1, new-feature, fix, etc. (non-descriptive and generic)

• Johns-branch, dev_branch, etc. (personal names or unclear purpose)

• BUGFIX, FeAtUrE, etc. (inconsistent casing)

Remember that these are general best practices, and you should adapt them to suit your team's workflow and preferences. Consistency within the team is key to ensuring everyone understands the branch naming conventions and follows them consistently.

htmx is a library that allows you to add dynamic behavior to your web pages using HTML attributes. It enables you to update parts of a web page without having to refresh the entire page or write a lot of JavaScript code. It works by sending small, targeted requests to the server and updating only the relevant parts of the page with the server's response.

Django and htmx can work together to create highly dynamic and interactive web applications. htmx can be used to add dynamic behavior to Django's HTML templates, such as updating a list of items without refreshing the page, or adding a new item to a form without having to navigate away from the page.

One popular htmx feature is its ability to easily handle form submissions with minimal JavaScript code. With htmx, you can submit a form using an HTTP request, and the server can return a partial HTML response that updates only the relevant parts of the page.

Setup

1. Include HTMX CDN in Django template.

    <script src="https://cdnjs.cloudflare.com/ajax/libs/htmx/1.8.6/htmx.min.js" integrity="sha512-SqV24lSRcthd1Bg0Obg/W/qWhKdSsQPIV342MQd8qDeeY/gJt+2qmsLhutblQNDuUx0Xs2Jh7D8m+7S1b0wQyA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
   

2. Install django-htmx for better handling of HTMX requests.

    pip install django-htmx
   

3. Next, you need to add 'django_htmx' to your INSTALLED_APPS list in your Django project's settings.py file:

    INSTALLED_APPS = [
        # ... other apps ...
        'django_htmx',
    ]
   

4. Include django_htmx.middleware.HtmxMiddleware in MIDDLEWARE list in your Django project's settings.py file.

    MIDDLEWARE = [
        # ... other middlewares ...
        "django_htmx.middleware.HtmxMiddleware",
    ]
   

5. Also, include {% django_htmx_script %} in django template.

    {% django_htmx_script %}
   

Now you're ready to use HTMX in templates.

Form submission

To submit a form using Django and htmx, you can follow these steps:

1. Add the hx-post attribute to the <form> tag in your HTML template. This will tell htmx to submit the form using an AJAX request instead of a full page reload. You can also add the hx-target attribute to specify where the server response should be inserted. For example:

    <form action="{% url 'my_view' %}" method="POST" hx-post hx-target="#form-container">
        {% csrf_token %}
        <!-- form fields go here -->
        <button type="submit">Submit</button>
    </form>
    <div id="form-container">
        <!-- server response will be inserted here -->
    </div>
   

2. In your Django view function, you can check if the request method is POST and handle the form submission accordingly. You can use the render function to render a template with the server response. For example:

    from django.shortcuts import render
   
    def my_view(request):
        if request.method == 'POST':
            # handle form submission
            # ...
            # render the response
            return render(request, 'response_template.html',    {'message': 'Form submitted successfully!'})
        else:
            # render the form
            return render(request, 'form_template.html')
   

3. Create a new HTML template for the server response, and include the response message or data. For example:

    <p>{{ message }}</p>
   

That's it! When the user submits the form, htmx will send an AJAX request to your Django view function, and the server response will be inserted into the specified target element without a full page reload.

Basic Commands

Here are some basic commands in Linux:

1. ls: Lists the files and directories in the current directory.

2. cd: Changes the current directory.

3. pwd: Prints the working directory.

4. mkdir: Creates a new directory.

5. rmdir: Removes a directory.

6. touch: Creates a new file.

7. rm: Removes a file.

8. cp: Copies a file.

9. mv: Moves or renames a file.

10. cat: Displays the contents of a file.

11. grep: Searches for a pattern in a file.

12. sudo: Executes a command as the root user.

13. apt-get: Installs, updates, or removes software packages.

14. chmod: Changes the permissions of a file or directory.

15. chown: Changes the ownership of a file or directory.

File System Commands

Here are some commands related to file system in Linux:

1. df: Displays the disk usage and available space on mounted file systems.

2. du: Displays the disk usage of a file or directory.

3. fdisk: A partitioning utility that can create, delete, and manipulate disk partitions.

4. mkfs: Formats a file system on a disk partition.

5. mount: Mounts a file system to a directory.

6. umount: Unmounts a file system from a directory.

7. blkid: Displays the UUID and file system type of all available block devices.

8. find: Searches for files and directories based on certain criteria such as name, size, or modification time.

9. locate: Searches for files and directories based on their names.

10. ln: Creates a hard or symbolic link to a file.

11. tar: Creates or extracts a compressed archive file.

12. rsync: Synchronizes the contents of two directories or files across a network.

13. diff: Compares the contents of two files and displays the differences.

14. grep: Searches for a pattern in a file or multiple files.

15. awk: A text processing tool that can extract and manipulate data from files.

Memory Management Commands:

Here are some commands related to memory management system in Linux:

1. free: Displays the amount of free and used memory in the system.

2. top: Displays real-time information about the system's processes and memory usage.

3. ps: Displays information about the currently running processes.

4. kill: Sends a signal to terminate a process.

5. nice: Adjusts the priority of a process to control its resource usage.

6. renice: Changes the priority of a running process.

7. vmstat: Displays virtual memory statistics such as page faults and swap usage.

8. swapon: Activates a swap partition or file.

9. swapoff: Deactivates a swap partition or file.

10. ulimit: Sets limits on system resources such as memory usage and file size for user processes.

11. top: Provides a dynamic real-time view of the processes running in a system, with detailed information about the memory and CPU usage of each process.

12. htop: A more advanced version of top, which provides additional functionality and an interactive interface.

13. pmap: Displays the memory map of a process, showing the virtual memory addresses used by each memory-mapped file or shared library.

14. sysctl: Configures kernel parameters related to memory management, such as swappiness or the size of the page cache.

15. numactl: Configures NUMA (non-uniform memory access) settings, such as the allocation policy for memory and CPUs.

In this blog post, we will explore some of the most useful psql commands that every PostgreSQL developer should know. From creating databases and tables to querying data and managing users, we'll cover a wide range of commands that will help you become more efficient and effective in working with PostgreSQL. Whether you're a beginner or an experienced PostgreSQL developer, this post will provide you with the knowledge you need to take your skills to the next level. So, let's dive in and discover some of the most essential psql commands!

1. \du: Lists all users in the current database and their roles.

2. \dt: Lists all tables in the current database.

3. \d <table_name>: Provides a description of a specified table, including the columns, their data types, and any constraints.

4. \i <file_name>: Executes the SQL commands contained in the specified file.

5. \e: Opens the default editor to edit the current command or query.

6. \timing: Toggles display of the query execution time after each query.

7. \! <shell_command>: Executes the specified shell command from within psql.

8. \conninfo: Shows information about the current database connection, including the username, database name, host, and port.

9. \x: Toggles expanded display mode, which shows query results in a more readable format.

10. \h: Lists all available psql commands with brief descriptions.

11. psql -d <database_name>: Connect to a database.

12. createdb <database_name>: Create a new database.

13. \l: List databases.

14. \q: Quit PSQL.

15. \connect <database_name> or \c <database_name>: Connect to a specific database.

Note that some of these commands have additional options and arguments that can be used to modify their behavior. You can find more information on these options and arguments in the PostgreSQL documentation.

Commonly used SQL Commands

1. Creating a Superuser

Using SQL (within psql or via a script):

-- Using CREATE ROLE:
CREATE ROLE my_superuser WITH LOGIN SUPERUSER PASSWORD 'your_secure_password';

-- Alternatively, using CREATE USER (they’re almost interchangeable in PostgreSQL):
CREATE USER my_superuser WITH SUPERUSER PASSWORD 'your_secure_password';

Replace my_superuser and 'your_secure_password' with your desired username and a strong, secure password.

Tip: On Unix-based systems, you can also create a superuser directly from the shell:

sudo -u postgres createuser --superuser my_superuser

2. Commonly Used psql Meta-Commands

When you’re inside the psql shell, these slash (\) commands are super useful:

• \l or \list: List all databases.

• \c [database_name]: Connect to a specific database.

• \du: List all roles (users).

• \d: List all database objects (tables, views, indexes, etc.).

• \dt: List only tables.

• \dv: List views.

• \df: List functions.

• \i [file_name.sql]: Execute commands from an external file.

• \?: Display help on all meta-commands.

• \q: Quit the psql session.

3. Common SQL Commands

Creating a Database and Table:

-- Create a new database:
CREATE DATABASE my_database;

-- Connect to the database (if not already connected, use psql's \c command):
\c my_database

-- Create a table:
CREATE TABLE my_table (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Manipulating Data:

-- Inserting Data:
INSERT INTO my_table (name) VALUES ('Alice'), ('Bob');

-- Querying Data:
SELECT * FROM my_table;

-- Updating Data:
UPDATE my_table SET name = 'Charlie' WHERE id = 1;

-- Deleting Data:
DELETE FROM my_table WHERE id = 2;

4. Extra Tips & Tricks

• Direct Connection via Command Line:
  Use this command to jump straight into a specific database:

    psql -U postgres -d my_database
  

• Scripting:
  Write your SQL commands in a file (e.g., script.sql) and run them with:

    \i script.sql
  

• Security First:
  Always safeguard your credentials and avoid hard-coding sensitive information in your scripts.

Each user on a Unix system can have their own crontab file, and each file can contain multiple jobs scheduled to run at different intervals. Crontab syntax consists of six fields separated by spaces: minute, hour, day of the month, month, day of the week, and the command to be executed.

Crontab is often used to do things like run backups, update databases, and take care of the system. It makes it easy for system administrators and developers to automate tasks that they do often and saves them time and effort.

Example

Here is an example of a crontab entry that runs a script every day at 2:30 AM:

30 2 * * * /path/to/script.sh

Now, let's break down the components of this crontab entry:

• 30: This is the minute field. It specifies that the script should be executed at the 30th minute of the hour.

• 2: This is the hour field. It specifies that the script should be executed at 2:30 AM.

• *: This is the day of the month field. It specifies that the script should be executed every day of the month.

• *: This is the month field. It specifies that the script should be executed every month.

• *: This is the day of the week field. It specifies that the script should be executed every day of the week.

• /path/to/script.sh: This is the command to be executed. It specifies the full path to the script that should be executed.

So, in summary, this crontab entry runs the specified script every day at 2:30 AM, regardless of the day of the week or the month.

Commands

Here are some basic crontab commands that you can use to manage your crontab:

• crontab -e: This command opens the crontab file in the default editor, allowing you to add or modify crontab entries.

• crontab -l: This command lists the current crontab entries for the current user.

• crontab -r: This command removes all crontab entries for the current user.

• crontab -u [username] -e: This command opens the crontab file for the specified user, allowing you to add or modify crontab entries for that user.

• crontab -u [username] -l: This command lists the crontab entries for the specified user.

These are some of the most commonly used crontab commands. There are many other options available, which you can find by running man crontab in your terminal.

Logs

Logging the output of a crontab job to a file can be useful for debugging and troubleshooting. You can do this by adding >> /path/to/logfile 2>&1 at the end of your crontab command. Here is an example:

30 2 * * * /path/to/script.sh >> /path/to/logfile 2>&1

This will redirect both standard output (stdout) and standard error (stderr) to the specified log file.

If you want to suppress the output of a crontab job entirely, you can add > /dev/null 2>&1 at the end of the command. Here is an example:

30 2 * * * /path/to/script.sh > /dev/null 2>&1

This will redirect both stdout and stderr to the null device, effectively discarding any output.

Extras

• Always test your crontab entries before scheduling them to run automatically. You can do this by running the command manually and verifying that it produces the desired output.

• Use absolute paths for all commands and files referenced in your crontab. This ensures that the correct files and commands are used, even if the current working directory is different from what you expect.

• Be mindful of time zones. By default, crontab runs commands in the server's time zone. If you need to schedule jobs in a different time zone, you can use the TZ environment variable to set the desired time zone.

• Avoid overlapping jobs. If you have multiple crontab jobs that run at the same time, make sure they don't interfere with each other or cause performance issues. Consider staggering the jobs or using a job queue to manage them.

• Use descriptive comments to document your crontab entries. This makes it easier to understand what each job does and why it's scheduled to run at a specific time.

Usage

With Django

To use crontab with Django management commands, follow these steps:

1. Open your crontab configuration file by typing the following command in your terminal:

crontab -e

1. In the crontab file, add a new line for the task you want to schedule, using the following syntax:

* * * * * /path/to/python /path/to/manage.py <django-management-command> [options]

Replace /path/to/python with the path to your Python executable, /path/to/manage.py with the path to your Django project's manage.py file, <django-management-command> with the Django management command you want to run, and [options] with any additional options you want to pass to the management command.

For example, if you want to run the my_custom_command management command every day at midnight, you would add the following line to your crontab file:

0 0 * * * /usr/bin/python /path/to/myproject/manage.py my_custom_command

1. Save and close the crontab file. Your scheduled task will now run automatically according to the schedule you specified in the crontab file.

Note that when using crontab with Django management commands, you may need to set the DJANGO_SETTINGS_MODULE environment variable to the path of your project's settings module. You can do this by adding the following line at the beginning of your crontab file:

DJANGO_SETTINGS_MODULE=myproject.settings

Replace myproject.settings with the path to your project's settings module.

project_name/
├── config/
│   ├── settings/
│   │   ├── base.py
│   │   ├── local.py
│   │   ├── development.py
│   │   ├── staging.py
│   │   ├── production.py
│   ├── urls.py
│   ├── wsgi.py
│   ├── asgi.py
│
├── apps/
│   ├── app1/
│   │   ├── migrations/
│   │   ├── templates/
│   │   ├── static/
│   │   ├── tasks/
│   │   ├── utils/
│   │   ├── management/
│   │   ├── api/
│   │   │   ├── v1/
│   │   │   │   ├── serializers.py
│   │   │   │   ├── views.py
│   │   │   │   ├── filters.py
│   │   │   ├── v2/
│   │   ├── __init__.py
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── choices.py
│   │   ├── models.py
│   │   ├── signals.py
│   │   ├── urls.py
│   │   ├── views.py
│   ├── app2/
│   ├── app3/│
│
├── static/
│   ├── css/
│   ├── js/
│   ├── img/
│
├── media/
│
├── templates/
│
├── utils/
│
├── tasks/
│
├── management/
│
├── manage.py
│

In this structure, the main project folder contains three main directories: config, apps, static, and media.

• config directory contains all the settings and configurations of the Django project. It includes settings for different environments such as base.py (base settings), local.py (local development settings), development.py (development environment settings), staging.py (staging environment settings), and production.py (production environment settings). It also includes urls.py (main project URL configuration), wsgi.py (Web Server Gateway Interface configuration), and asgi.py (Asynchronous Server Gateway Interface configuration).

• apps directory contains all the apps that are part of the Django project. Each app has its own directory containing models, views, templates, static files, and tests. This structure helps to keep the code organized and makes it easier to manage.

  • tasks/: This directory contains files for background tasks such as Celery tasks.

  • templates/: This directory contains all of the app's templates.

  • static/: This directory contains all of the app's static files, such as CSS, JavaScript, and images.

  • utils/: This directory contains utility files that are used across multiple apps in the project, such as common helper functions, mixins, or abstract classes.

  • management/: This directory contains custom management commands for the Django project. These commands can be used to automate tasks, create database entries, and perform other administrative functions.

  • api/: This directory is inside each app and contains versioned subdirectories for the API views. Each versioned subdirectory has its own serializers.py, filters.py and views.py files.

    • v1/ and v2/: These directories are inside each app's api/ directory and represent version 1 and version 2 of the API. Each versioned subdirectory can have its own serializers.py, filters.py and views.py files. which allows for different versions of the API to have different serializers and filters.

• static directory contains all the static files used in the project such as CSS, JavaScript, and images.

• media directory contains all the media files uploaded by the users of the website.

• templates directory contains all the HTML templates used in the project.

• manage.py is the entry point to the Django project and is used to execute various Django commands such as starting the development server, running database migrations, etc.

This is just a basic structure, and depending on the size and complexity of your project, you may need to modify it.

Container package all the dependencies and code your app needs to run into a single file, which will run the same way on any machine.

Terminologies

Docker Container

A container is a standard unit of software that packages up code and all its dependencies, so the application runs quickly and reliably from one computing environment to another.

Container Image

A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.

How containers work ?

By using the low-level mechanics of the host operating system, containers provide most of the isolation of virtual machines at a fraction of the computing power.

Virtual Machines

VMs are great at providing full process isolation for applications: there are very few ways a problem in the host operating system can affect the software running in the guest operating system, and vice-versa.

Cons of Virtual Machines

However, the isolation comes at a high cost: the computational overhead associated with virtualizing hardware for use by a guest OS is significant.

Image vs Container

Docker Daemon

• The background service running on the host that manages building, running and distributing Docker services.

• The daemon is the process that runs in the operating system which clients talk to.

Docker Client

• The command line tool that allows the use to interact with the daemon.

• More generally, there can be other forms of clients too - such as Kitematic which provide a GUI to the users.

Docker Hub

• A registry of Docker images.

• You can think of the registry as a directory of all available Docker images.

• If required, one can host their own Docker registries and use them for pulling images.

Docker Shell Commands

Run command

1. docker run hello-world

   • If the image of the name ‘hello-world’ is not present in the local registry, it will be pulled from the Docker Hub.

   • By default, it pulls the image fo the latest version from the Docker Hub.

   • pull command pulls the image, and run command does (pull + execute)

2. docker run -it busybox bash

   • bash is the command to run.

   • Busybox is a software suite that provides several Unix utilities.

   • -i flag makes the container interactive STDIN mode.

   • -t provides pseudo TTY to the container.

3. docker run busybox echo "Hello World"

   • echo is a command to print the text.

   • "Hello World" is the text to print.

   • If you’ve noticed, all of that happened pretty quickly. Imagine, booting up a virtual machine, running a command and then killing it. Now we know why they say containers are fast.

List Containers

1. docker ps

   • Lists all the running containers.

2. docker ps -a

   • Lists all the containers.

3. docker ps -q

   • Lists only the container IDs.

Stop Container

1. docker stop <container-id>

   • Stops the container with the given ID.

2. docker stop $(docker ps -q)

   • Stops all the running containers.

Remove Container

1. docker rm <container-id>

   • Removes the container with the given ID.

2. docker rm $(docker ps -q)

   • Removes all the running containers.

List Images

1. docker images -a

   • Lists all the images.

2. docker images -q

   • Lists only the image IDs.

Remove Image

1. docker rmi <image-id>

   • Removes the image with the given ID.

2. docker rmi $(docker images -q)

   • Removes all the images.

List Volume

1. docker volume ls

   • Lists all the volumes.

2. docker volume ls -q

   • Lists only the volume IDs.

Remove Volume

1. docker volume rm <volume-id>

   • Removes the volume with the given ID.

2. docker volume rm $(docker volume ls -q)

   • Removes all the volumes.

Delete all Containers

1. docker rm -f $(docker ps -a -q)

   • Removes all the containers.

Dockerfile

• A Dockerfile is a text file that describes how to build a Docker image.

• It contains all the commands a user could call on the comand line to assemble a Docker image.

Commands

1. FROM <image>

   • The FROM command specifies the base image that the Dockerfile will build on.

   • The image can be a Docker image name, a Docker image tag, or a Docker image ID.

   • The FROM command is required.

   • Eg: FROM ubuntu:latest

     • In this case, the Dockerfile will build on the latest version of the Ubuntu image.

2. MAINTAINER <name>

   • The MAINTAINER command specifies the person or organization that built the image.

   • The MAINTAINER command is optional.

   • Eg: MAINTAINER "Arjun Adhikari"

     • In this case, the Dockerfile will be built by Arjun Adhikari.

3. RUN <command>

   • The RUN command specifies a command that will be run when the image is built.

   • The RUN command is optional.

   • Eg: RUN apt-get update

     • In this case, the Dockerfile will run the apt-get update command.

   • Eg: RUN ["<executable>", "<arguments>"]

     • In this case, the Dockerfile will run the executable with the arguments.

     • Shell form, the command is run in a shell, which by default is /bin/sh -c on Linux or cmd /S /C on Windows.

4. CMD <command>

   • The main purpose of a CMD is to provide defaults for an executing container.

   • Eg: CMD ["<executable>", "<arguments>"]

     • In this case, the Dockerfile will execute the executable with the arguments.

5. ENTRYPOINT <command>

   • The ENTRYPOINT command specifies a command that will be run when the container is started.

   • If the ENTRYPOINT isn’t specified, the default command is /bin/sh -c.

   • However, if you want to override some of the system defaults, you can specify own entrypoint and therefore manipulate the environment.

   • The ENTRYPOINT command also allows arguments to be set from the Docker RUN command as well, whereas CMD cannot.

   • Eg: ENTRYPOINT ["<executable>", "<arguments>"]

     • In this case, the Dockerfile will execute the executable with the arguments.

6. WORKDIR <path>

   • The WORKDIR command specifies the working directory for the container.

   • Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY, and ADD instructions that follow it.

   • It can be used multiple times in the one Dockerfile.

   • If a relative path is provided, it will be relative to the path of the previous WORKDIR command.

   • Eg: WORKDIR /home/user

     • In this case, the Dockerfile will be built in the /home/user directory.

7. LABEL <key>=<value>

   • The LABEL command specifies a label for the image.

   • It is mainly used for setting metadata for the images.

   • It can be used multiple times in the one Dockerfile.

   • Eg: LABEL version=1.0

     • In this case, the Dockerfile will be labeled with the version 1.0.

8. EXPOSE <port>

   • The EXPOSE command specifies a port that the container will expose to the host.

   • Informs Docker that the container will be listening on the specified port at runtime.

   • EXPOSE doesnot make the ports of the container accessible to the host.

   • It can be used multiple times in the one Dockerfile.

   • Eg: EXPOSE 8080

     • In this case, the Dockerfile will expose the port 8080.

9. ENV <key>=<value>

   • The ENV command specifies an environment variable for the container.

   • It can be used multiple times in the one Dockerfile.

   • Eg: ENV PATH=/usr/local/bin

     • In this case, the Dockerfile will set the PATH environment variable to /usr/local/bin.

10. ADD <source> <destination>

• The ADD command specifies a file or directory that will be copied into the container at runtime.

• Eg: ADD /home/user/file.txt /home/user/file.txt

  • In this case, the Dockerfile will copy the file.txt from the host to the container.

1. COPY <source> <destination>

• The COPY command specifies a file or directory that will be copied into the container at runtime.

• Eg: COPY /home/user/file.txt /home/user/file.txt

  • In this case, the Dockerfile will copy the file.txt from the host to the container.

COPY only supports the basic copying of local files into the container, while ADD has some features (like local-only tar extraction and remote URL support) that are not immediately obvious. Consequently, the best use for ADD is local tar file auto-extraction into the image, as in ADD rootfs.

Example - Lint Python Code from a Repository

FROM ubuntu:latest

WORKDIR /workspace


RUN apt-get update && apt-get install -y \
    git python3-pip

RUN git clone https://github.com/theArjun/python-codes 
RUN pip3 install pylint
RUN pylint python-codes

Here, I am going to mention some anti-patterns in Django models.

1. Using len(queryset) instead of queryset.count()

   • The queryset in Django are lazily evaluated which means that records in database aren’t read from database until we interact with the data.

   • len(queryset) performs the count of database records by Python interpreter in application level. For doing so, all the records should be fetched from the database at first, which is computationally heavy operation.
     Whereas, queryset.count() calculates the count at the database level and just returns the count.

For a model Post :

    from django.db import models

    class Post(models.Model):
        author = models.CharField(max_length=100)
        title = models.CharField(max_length=200)
        content = models.TextField()

If we use len(queryset), it handles the calculation like SELECT * FROM post which returns a list of records (queryset) and then python interpreter calculates the length of queryset which is similar to list data structure. Imagine the waste in downloading many records only to check the length and throw them away at the end! But, if we need the records after reading the length, then len(queryset) can be valid.

If we use queryset.count(), it handles the calculation like SELECT COUNT(*) FROM post at database level. It makes the code execution quicker and improves database performance.

1. Using queryset.count() instead of queryset.exists()

   • While we kept praising the use of queryset.count() to check the length of a queryset, using it may be performance heavy if we want to check the existence of the queryset.
     For the same model Post, when we want to check if there are any post written by author Arjun, we may do something like:

    posts_by_arjun: Queryset = Post.objects.filter(author__iexact='Arjun')

    if posts_by_arjun.count() > 0:
        print('Arjun writes posts here.')
    else:
        print('Arjun doesnt write posts here.)

The posts_by_arjun.count() performs an SQL operation that scans every row in a database table. But, if we are just interested in If Arjun writes posts here or not ? then, more efficient code will be:

    posts_by_arjun: Queryset = Post.objects.filter(author__iexact='Arjun')

    if posts_by_arjun.exists():
        print('Arjun writes posts here.')
    else:
        print('Arjun doesnt write posts here.)

posts_by_arjun.exists() returns a bool expression that finds out if at least one result exists or not. It simply reads a single record in the most optimized way (removing ordering, clearing any user-defined select_related() or distinct() methods.)

Also, checking existence / truthiness of queryset like this is inefficient.

    posts_by_arjun: Queryset = Post.objects.filter(author__iexact='Arjun')

    if posts_by_arjun:
        print('Arjun writes posts here.')
    else:
        print('Arjun doesnt write posts here.)

This does the fine job in checking if there are any posts by Arjun or not but is computationally heavy for larger no of records. Hence, use of queryset.exists() is encouraged for checking existence / truthiness of querysets.

1. Using signals excessively

   • Django signals are great for triggering jobs based on events. But it has some valid cases, and they shouldn’t be used excessively. Think of any alternative for signals within your codebase, brainstorm on its substitution and try to place signals logic in your models itself, if possible.

   • They are not executed asynchronously. There is no background thread or worker to execute them. If you want some background worker to do your job for you, try using celery.

   • As signals are spread over separate files if you’re working on a larger project, they can be harder to trace for someone who is a fresh joiner to the company and that’s not great. Although, django-debug-toolbar does some help in tracing the triggered signals.

Let’s create a scenario where we want to keep the record of Post writings in a separate model PostWritings.

    class PostWritings(models.Model):
        author = models.CharField(max_length=100, unique=True)
        posts_written = models.PositiveIntegerField(default=0)

If we want to automatically update the PostWritings record for a use based on records created on Post model, there are ways to achieve the task with / without signals.

A. With Signals

    from django.db.models.signals import post_save
    from django.db.models import F
    from django.dispatch import receiver
    from .models import Post

    @receiver(sender=Post, post_save)
    def post_writing_handler(sender, instance, created, **kwargs):
    if created:
        writing, created = PostWritings.objects.get_or_create(author=instance.author)
        writing.update(posts_written=F('posts_written') + 1)

B. Without Signals

We need to override the save() method for Post model.

    from django.db import models
    from django.db.models import F

    class Post(models.Model):
        author = models.CharField(max_length=100)
        title = models.CharField(max_length=200)
        content = models.TextField()

        def save(self, *args, **kwargs:
            # Overridden method.
            author = self.author
            if self.id:
                writing, created = PostWritings.objects.get_or_create(author=author)
                writing.update(posts_written=F('posts_written') + 1)
            super(Post, self).save(*args, **kwargs)

As the same job can be accomplished without signals, the code can be easily traced and prevent unnecessary event triggers.
If someone feels about not having readability on save() method here, breaking up code is always great. Let’s do that.

    from django.db import models
    from django.db.models import F

    class Post(models.Model):
        author = models.CharField(max_length=100)
        title = models.CharField(max_length=200)
        content = models.TextField()

        def _update_post_writing(self, created=False, author=None):
            if author is not None and craeted:
                writing, created = PostWritings.objects.get_or_create(author=author)
                writing.update(posts_written=F('posts_written') + 1)

        def save(self, *args, **kwargs:
            # Overridden method.
            author = self.author
            created = self.id is None
            super(Post, self).save(*args, **kwargs)
            self._update_post_writing(created, author)

1. Not defining an __str__ method for a model

   When you define a model, it is important to create a string representation of that model so that it can be displayed accurately. If the __str__ method is not defined, then it will not be possible to access the model and display it within the user interface correctly.

2. Not using the built-in ModelForm validation

   Using the built-in ModelForm validation can help to ensure that the data being entered into the form is valid. This is useful for form validation, preventing invalid data from being entered and ensuring that forms are processed correctly.

3. Having complex relationships or calculations within your model

   Models should not contain too much complexity as this can lead to them becoming difficult to maintain. Keeping the relationships and any calculations within the model to a minimum helps to keep things organized and makes it easier to debug any potential issues.

4. Not using get_or_create or bulk_create

   When creating large numbers of objects - When creating large numbers of objects, it is important to use the get_or_create or bulk_create methods rather than creating them one at a time. This can help to make the process faster and more efficient by avoiding unnecessary extra steps.

5. Ignoring the unique_together or unique_for_date fields

   When creating model instances - It is important to pay attention to the unique_together and unique_for_date fields when creating model instances, as they are used to ensure that data is unique and consistent. Failing to take these fields into account can lead to duplicate data being written to the database.

We appear to have learned how to mitigate some Django Model Anti Patterns. For now, thank you everyone for having me here. I'll be back with more Django-related content soon. You can also find me on GitHub. Till then keep coding :)

Introduction

In this blog, I am guiding to integrate django-jazzmin theme to the Django project. According to their documentation :

Django Jazzmin is intended as a drop-in app to jazz up your Django admin site, with plenty of things you can easily customise, including a built-in UI customizer.

The default admin dashboard provided by Django isn’t best fitted for our need every time and we need to customize a lot to match the theme with our need. And some of us may not prefer to go with the default theme which looks like this :

For the developers who want to get a customized and better-looking admin dashboard for Django, django-jazzmin is the one with better UI components with features like :

• Drop-in admin skin, all configuration optional

• Select2 drop-downs

• Bootstrap 4 & AdminLTE UI components

• Search bar for any given model admin

• Modal windows instead of popups

• Customisable side menu

• Customisable top menu

• Customisable user menu

• Responsive

• Customisable UI (via Live UI changes, or custom CSS/JS)

• Based on the latest adminlte + bootstrap

which looks like this:

To achieve a good looking AdminLTE dashboard on our Django project, we need to follow a sequence of instructions.

Instructions

Navigate to your workspace:

# Terminal
# =========

$ mkdir - ~/workspace/integrate-django-jazzmin
$ cd ~/workspace/integrate-django-jazzmin

Setup Virtual Environment (Optional)

# Terminal
# =========

$ python -m virtualenv venv

created virtual environment CPython3.8.6.final.0-64 in 116ms
  creator CPython3Posix(dest=/home/ubuntu/workspace/integrate-django-jazzmin/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

This will set up a virtualenv named venv which isolates libraries and dependencies from the global installation. Now, activate the virtual environment.

$ source venv/bin/activate

Installing libraries

Now, we are good to install the libraries we need to set up the Django project.

$ pip install django django-jazzmin

This will install the django and django-jazzmin library.

Create Django Project

It’s time to create a Django project.

$ django-admin startproject dj_jazzmin_demo .

For a better demonstration of the theme, I’m going to create a Django app named blog.

$ python manage.py startapp blog

which created directory structure like this:

# Directory Structure
# ====================
.
├── blog
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── dj_jazzmin_demo
│   ├── asgi.py
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── settings.cpython-36.pyc
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

Register Apps

Now, let’s register the blog app inside INSTALLED_APPS in settings.py. Add blog.apps.BlogConfig after the pre-registered django apps. Also, we are good to add our theme to the django project. For that, register jazzmin theme to INSTALLED_APPS. But, we’re registering jazzmin on the top of other pre-registered apps because we want the theme to load earlier than other apps.

# dj_jazzmin_demo/settings.py
# =============================

INSTALLED_APPS = [
    # Pre-loading Third Party apps
    'jazzmin', 

    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    # Local Apps
    'blog.apps.BlogConfig',

]

Create Model and ModelAdmin

Now, we have our app and theme registered in our project. Then, we can write models for the app blog. Let’s write a model named Post in models.py inside blog app.

# blog/models.py
# ================

from django.db import models
from django.utils.translation import ugettext_lazy as _

class Post(models.Model):
    '''Model definition for Post.'''

    title = models.CharField(_('Post'), max_length=50)
    content = models.TextField(_('Content'))
    is_publishable = models.BooleanField(_('Is Publishable ?'), default=False)
    created_at = models.DateTimeField(_('Created at '),auto_now_add=True)
    updated_at = models.DateTimeField(_('Updated at '),auto_now=True)

    class Meta:
        '''Meta definition for Post.'''

        verbose_name = 'Post'
        verbose_name_plural = 'Posts'

    def __str__(self):
        '''Unicode representation of Post.'''
        return self.title

For making our app accessible via the admin dashboard, we’re going to register the model Post in admin.py.

# blog/admin.py
# ==============

from django.contrib import admin
from .models import Post


@admin.register(Post)
class PostAdmin(admin.ModelAdmin):
    '''Admin View for Post'''

    list_display = (
        'title',
        'is_publishable',
        'created_at',
        'updated_at',
    )
    list_filter = (
        'is_publishable',
        'created_at',
        'updated_at',
    )

Make Migrations and Migrate Models

Now, we’re good to make migrations and migrate the model we created.

# Terminal
# =========

$ python manage.py makemigrations blog
Migrations for 'blog':
  blog/migrations/0001_initial.py
    - Create model Post

# Terminal
# =========

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, blog, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying blog.0001_initial... OK
  Applying sessions.0001_initial... OK

For accessing the admin panel, we need to create a superuser which has the privilege to access the admin dashboard and perform operations over there.

# Terminal
# =========

$ python manage.py createsuperuser
Username (leave blank to use 'ubuntu'): ubuntu
Email address: 
Password: 
Password (again): 
Superuser created successfully.

Enjoy the output

Finally, we’ve completed the procedure to create a basic Django project and integrate django-jazzmin theme. Now, we can run our django project.

# Terminal
# =========

$ python manage.py runserver

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
November 13, 2020 - 19:46:51
Django version 3.1.3, using settings 'dj_jazzmin_demo.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C

We have our Django project running. To access it through a web browser, visit the URL http://127.0.0.1:8000/admin/ on a web browser. It loads to a different login interface than default login UI provided by Django.

After entering credentials, we’re navigated to a jazzy looking admin interface provided by django-jazzmin.

For now, we’ve successfully integrated django-jazzmin theme to our django_project.

Thanks for reading.

In this blog, I'll show you how to add autocomplete to the Django admin interface. I chose the admin interface because it lets me create, read, update, and delete (CREATE, READ, UPDATE, DELETE) models.

For implementing the auto completion feature, I am going to use django-ajax-selects currently maintained by crucialfelix.

Setup

Let's begin by making a Django project. First of all, I’ll create a virtual environment .venv. You can create a virtual environment just by python -m venv .venv. Let’s activate the virtual environment .venv\Scripts\activate as I am currently on Windows.

After that, install django and django-ajax-selects using pip.

pip install django django-ajax-selects

Following installing libraries, let’s create django project named autocomplete_demo_project.

django-admin startproject autocomplete_demo_project

Subsequently, Let’s create a django app named blog .

python manage.py startapp blog

And register the app 'blog' and 'ajax_select' in INSTALLED_APPS inside settings.py.

# autocomplete_demo_project/settings.py
INSTALLED_APPS  = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'ajax_select', # new apps
    'blog.apps.BlogConfig',
]

Thereafter, let’s define URLConfs for ajax_select so it may load the necessary javascripts, stylesheets for autocompletion.

# autocomplete_demo_project/urls.py
from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static
from ajax_select import urls as ajax_select_urls

urlpatterns = [
    path('ajax_select/', include(ajax_select_urls)),
    path('admin/', admin.site.urls),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

Models and Lookups

We have to start by defining models which will later be used for observing auto-completion in admin panel. Without adding any complexity, I’ll just start with a single field name in Blog model.

# blog/models.py
from django.db import models
from django.utils.translation import gettext_lazy as _

class Blog(models.Model):
    name = models.CharField(_("Blog"), max_length=50)

def  __str__(self):
    return  self.name

Let’s also create a lookup for the field name which fetches the most suitable result by matching the query entered while entering a record in admin panel. For doing so, let’s create LookupChannel for our field name (NameLookup) in lookup.py inside blog app directory.

# blog/lookups.py
from ajax_select import register, LookupChannel

from .models import Blog

@register('names')
class  NameLookup(LookupChannel):
    model = Blog

    def  get_query(self, q, request):
        return  self.model.objects.filter(name__icontains=q).order_by('name')[:50]

    def  format_item_display(self, item):
        return  u"<span class='blog_name'>%s</span>"  % item.name

Here, we’ve observed two methods inside our NameLookup class. Let’s dig into both of them:

1. get_query method takes q parameter as query which will be used to filter the record in database. Here, we’ve used the name__icontains approach which will filter the matched records in DB with sub string query q and return the filtered records. We may also proceed with name__startswith approach which only tries to match the starting string with the records in DB with sub string query q and return the filtered records. But, let’s just stick into one approach ( name__icontains approach) to implement the auto completion.

2. format_item_display methods takes each filtered records and displaysblog.name (data) which is the need in our case with proper styling. We can override the display properties to make it look better and match with our preferences.

Also notice that we’ve registered a lookup name names with register decorator which will later be used with form to match with required form field i.e. The form field with the property names will match with the above lookup.

ModelAdmin and Forms

Ensuing step will be to create Django forms which will be provided as property to our ModelAdmin which we’ll later create in admin.py .

# blog/forms.py
from ajax_select.fields import AutoCompleteField
from django import forms

from .models import Blog

class  BlogForm(forms.ModelForm):

    name = AutoCompleteField('names')
    class  Meta:
        model = Blog
        fields = [
            'name'
        ]

It’s worth noticing that the field name is an AutoCompleteField with the property names which is a lookup name we’ve registered in NameLookup class inside blog/lookups.py. It’s how we interlink forms and lookups using django-ajax-selects.

Following this, let’s create ModelAdmin inside admin.py.

# blog/admin.py
from ajax_select.admin import AjaxSelectAdmin
from django.contrib import admin

from .forms import BlogForm
from .models import Blog

@admin.register(Blog)
class  BlogAdmin(AjaxSelectAdmin):

    form = BlogForm

What we’ve done here is overriding the default form property provided by Django Admin and get it replaced by our BlogForm form object.

Also note that, the use of BlogForm isn’t limited to ModelAdmin for auto completion. It can also be used in used defined templates by passing form inside context parameter from views.py. For doing so,

{{ form.media }}
{{ form }}

We need to load ((form.media}} to load the necessary JS and CSS files defined by ajax_selects app.

Observation

After defining lookups, models, forms and modeladmin, now we can make migrations and run the migration files.
python manage.py makemigrations blog

Post hoc, let’s run those migration files.
python manage.py migrate

Then, create a superuser for accessing admin interface,
python manage.py createsuperuser

Then, log into 127.0.0.1:8000/admin to access admin interface and navigate to Blog model.

Try adding some records and you’ll notice auto completion making work easy for you.

Demo

I’ve added some records in my models.

Let’s try to add another record to observe auto-completion.

Notice that, when I am typing A, the records with the letter A in their name appeared as a dropdown so the user may autocomplete the record.

Thank you for reading.

Navigate the repository and directory you want

First of all, navigate to the repository from which you want to access the folder. Here, I am taking my Machine Learning repository as an example.

Then, I am navigating to the directory named data-visualization which I want to download. Then copy the URL from the browser address bar which is https://github.com/theArjun/machine-learning/tree/master/data-visualization in my case. Note this or get it stored in the clipboard.

Install Subversion

I am using subversion to download the required directory. Make sure you have it installed on your system before proceeding.
apt-get install subversion

Tweak URL

I hope you’ve noted the directory URL from GitHub or got it stored on your clipboard, which was https://github.com/theArjun/machine-learning/tree/master/data-visualization in my case. Now replace the tree/master with trunk which results in https://github.com/theArjun/machine-learning/trunk/data-visualization and note it again. Good going, pals, We’re now at the final stage.

Download the Directory

Launch the terminal and type the following command:
svn checkout https://github.com/theArjun/machine-learning/trunk/data-visualization

The URL was the tweaked URL that I instructed in the Tweak URL section. The execution of the command looks like the following in the terminal.

$ svn checkout https://github.com/theArjun/machine-learning/trunk/data-visualization

A    data-visualization/Data Generation - Classification.ipynb
A    data-visualization/Data Generation - Regression Data.ipynb
A    data-visualization/Data Normalization and Standardization.ipynb
A    data-visualization/Data Visualization - Subplots.ipynb
A    data-visualization/Exploring Sklearn Datasets.ipynb
A    data-visualization/Generating Data using Sk-Learn.ipynb
A    data-visualization/Matplotlib Basics.ipynb
A    data-visualization/Multivariate Normal Distribution  .ipynb
A    data-visualization/Normal Distribution & Histogram.ipynb
A    data-visualization/Numeric Spacing.ipynb
A    data-visualization/PyCharts, BarCharts and Legends.ipynb
A    data-visualization/Template.ipynb
A    data-visualization/Working with Pandas - MNIST Dataset.ipynb
A    data-visualization/Working with Pandas.ipynb
A    data-visualization/csv
A    data-visualization/csv/linearX.csv
A    data-visualization/csv/linearY.csv
A    data-visualization/csv/movie_data.csv
Checked out revision 103.

and a directory named data-visualization gets created in the working folder.

$ ls

data-visualization

This folder contains all the files that are in the data-visualization folder on GitHub, i.e., files that are synced with the remote (GitHub).

So, no worries, create as many folders as you need and download them as required without cloning the whole bulky repository.

Thanks for reading.