Python Wizardry by AppSignal

Monitoring Django Query Performance with AppSignal

Roy Tomeij — 2026-04-16T00:00:00+00:00

Slow database queries are really a pain. It’s easy to blame Django for taking ages to process a request, but the real issue may lie in an SQL query not doing what it should be. These performance degrading queries are often hiding in plain sight. By the time you notice them, they are already affecting your end users.

AppSignal is an application performance monitoring (APM) tool with support for Python and Django out of the box. It can surface these exact issues with detailed insights for each individual request. In this short guide, you will learn how to quickly install it in your project. You will also find out about some common bad practices when querying in Django.

Let's get this thing going!

Prerequisites

Before you get started, you should have:

A Django project up and running
A seeded database of your choice (MySQL, PostgreSQL, SQLite3)
An application set up on AppSignal

Setting Up AppSignal in a Django Project

Let's start off by setting up AppSignal in your existing Django project. Navigate to the root of your project's directory and add the following packages inside the requirements.txt file:

appsignal: the AppSignal library
opentelemetry-instrumentation-django: automatically instruments Django's request/response cycle, capturing each request as a trace
opentelemetry-instrumentation-wsgi: instruments the WSGI layer so that spans are created at the server level before Django's middleware even runs
opentelemetry-instrumentation-asgi: the async equivalent of the above, required if you're running Django with an ASGI server like Uvicorn or Daphne

# requirements.txt
appsignal
opentelemetry-instrumentation-django
opentelemetry-instrumentation-wsgi
opentelemetry-instrumentation-asgi

Based on the SQL database your application uses, you need to install Django ORM-specific instrumentation:

Note: The AppSignal for Python integration will automatically use this instrumentation when the corresponding package is installed. To disable this instrumentation without uninstalling the package, use the disable_default_instrumentations configuration option.

Next, install the packages and run the AppSignal's automated install tool, which will create __appsignal__.py for you in the project:

pip install -r requirements.txt
python -m appsignal install

Note: If the pip and python executables do not work, please use pip3 and python3 instead.

The tool will prompt you for an app name (previously created on AppSignal), as well as the Push API key, which can be found in the Push & deploy section of your App settings.

Django requires a bit more configuration. Integrate the manage.py file, and import the appsignal module needs in the manage.py file. Then, inside the main method, call appsignal.start to initialize instrumentation configured in the __appsignal__.py (done automatically by the automated install tool).

# manage.py
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys

# Add this import statement
import appsignal

def main():
    """Run administrative tasks."""
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<YOUR_APP_NAME_HERE>.settings')

    # Add this method call
    appsignal.start()

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)

if __name__ == '__main__':
    main()

Make sure to put your actual application name instead of the <YOUR_APP_NAME_HERE>.

Note: See more instructions for production setups using WSGI or ASGI.

The __appsignal__.py contains sensitive credentials (in this case the Push API key). Please do not commit this key to version control. Move this to your app's security credentials or environment variables. See more details.

That's it for the AppSignal setup! Quick and easy, right? Your app is ready to be monitored.

Next, let’s see how AppSignal tracks and traces database queries.

How AppSignal Tracks Database Queries

Once the server runs, AppSignal wraps Django's database backend. Every SQL statement your ORM or raw connection.cursor() generates becomes a span: a timed unit of work attached to the request that triggered it.

AppSignal breaks each request down into two components you'll see throughout the dashboard:

sql — time spent waiting on the database
django — time spent in Python (view logic, serialization, middleware)

In a healthy request, those numbers are small and balanced. When something goes wrong, one of them dominates. Let's see what kind of issues you can catch.

Identifying Slow Queries in the AppSignal Dashboard

First, you need to see the issues that have happened. Navigate to Performance > Issue list in your AppSignal app.

Here, you have an overview of:

The originating request (Action name)
Its severity and status
Mean time-to-completion, throughput, and impact on performance

By clicking on the originating request, you will open a full breakdown with all the details.

The Samples table shows the latest fetches, their durations, N+1 query detection, and group breakdown.

Press on a specific timestamp next.

This section shows the most intriguing details. The Sample breakdown shows how much time has elapsed for each part of the request, for sql, and django, that is. This can help you pinpoint what you need to address. Is it the server or the query?

AppSignal also associates locally available tags to the request. A waterfall timeline is also shown so you can see what happens in your request chronologically. Plus, this shows what query was performed at each step. Neat!

Common Django Query Problems and How to Spot Them

Now that AppSignal is surfacing your query data, you need to know what you're looking for. Most Django performance problems fall into a handful of recurring patterns. Here are the most common ones and how the AppSignal dashboard helps you catch them.

The Good Old N+1 Queries

College peeps learn this early on. This is the most frequent database query problem that sneaks up on you more easily that you’d think. In fact, N+1 happens when you fetch a list of objects and execute an additional query for each one.

# views.py
def book_list(request):
    books = Book.objects.all()  # 1 query
    for book in books:
        print(book.author.name)  # 1 query per book → N+1
    ...

The AppSignal Sample breakdown shows N+1 pattern as a long stack of near-identical sql calls in the waterfall timeline. The identical SELECT statement is repeated tens or even hundreds of times, each with a tiny duration that really adds up to a significant total.

AppSignal flags these directly in the Samples table with its built-in N+1 detection badge, so you do not have to spot the pattern yourself.

To fix the N+1 mishaps, you can almost always use select_related (for foreign keys and one-to-one relations) or prefetch_related (for many-to-many and reverse foreign keys):

books = Book.objects.select_related("author").all()

Missing Database Indexes

Querying a small dataset usually does not cause issues. But, when data grows, a suboptimal query will definitely become a bottleneck. This happens when columns being filtered or ordered are not indexed. Django will complete the query without complaining, but database just has to scan the entire table to give a response.

# Filtering on a column with no index becomes a full table scan
orders = Order.objects.filter(status="pending").order_by("created_at")

The fix is to add an index on the relevant column. In Django, this is done directly in the model:

class Order(models.Model):
    status = models.CharField(max_length=20, db_index=True)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        indexes = [
            models.Index(fields=["status", "created_at"])
        ]

If you frequently filter on multiple columns together, a composite index (as shown above) is more efficient than two separate single-column indexes.

Unbounded QuerySets

Fetching more rows than you actually need is another way to degrade performance. A view that calls Model.objects.all() without any limit will happily load every row in the table into memory. As the table grows, so does the response time and memory footprint.

# Fine with 500 rows. A disaster with 500,000.
def export_view(request):
    records = AuditLog.objects.all()
    ...

The fix depends on the use case. For paginated views, use Django's built-in Paginator or slice the QuerySet directly:

recent_logs = AuditLog.objects.order_by("-created_at")[:100]

Repeated Identical Queries

Sometimes the same query is executed multiple times within a single request, not as part of an N+1 loop, but because different parts of the codebase independently ask for the same data. Django's ORM does not cache QuerySet results across calls, and each evaluation hits the database.

Next Steps and Resources

Complex SQL queries can give you headaches, but queries that you cannot measure or understand will give you migraines. Now, if we stick with the same metaphor, using a tool like AppSignal to monitor direct SQL queries and Django calls can be like popping some ibuprofen.

In this guide you’ve learned how to install AppSignal within your Django application, as well as what you can expect to get from AppSignal when performing the queries. From here, you may want to try setting up triggered alerts or uptime monitoring.

AppSignal is free to start with and can be integrated within minutes in your project. Start spotting those pesky queries in no time!

Tracing a Slow Request Through Your Django App

Roy Tomeij — 2026-04-14T00:00:00+00:00

Slow endpoints are difficult to detect because they don’t fail. They simply get slower and slower. Average latency may look fine, but that can be misleading.

That’s why we need to look at other values, like p90 and p95, which often reflect what’s really going on. For example, p90 represents the slowest 10% of requests, and p95 represents the slowest 5%. When these values increase, users start experiencing delays.

For a backend developer, the problem is not necessarily how to make things faster; it’s rather how to identify where time is being spent. A slow request is not generally caused by one big query but by a series of small operations that add up.

In this article, we are going to trace a slow request using a small demo application built with Django. We’ll use AppSignal to identify why it’s slow and how we can improve it.

Setting up the Demo Application

We'll start with a simple project named django_books_demo. It’s a data store for books, along with their authors and publishers.

To create the project and an app, use the following commands:

django-admin startproject django_books_demo
cd django_books_demo
python manage.py startapp inventory

Here’s what the project structure should look like:

django_books_demo/
│
├── manage.py
├── django_books_demo/
│   ├── settings.py
│   ├── urls.py
│   └── ...
│
└── inventory/
    ├── models.py
    ├── views.py
    └── migrations/

Installing AppSignal

Next, we install the AppSignal Python integration:

pip install appsignal

Add the following to your Django settings:

APPSIGNAL_PUSH_API_KEY="YOUR_APPSIGNAL_KEY"
APPSIGNAL_APP_NAME="django-books-demo"
APPSIGNAL_ENVIRONMENT="development"

Now, we need to make sure the AppSignal agent is running when Django is running. We do this by adding the following to manage.py:

import appsignal

appsignal.start()

Once you have done this, AppSignal will automatically monitor the performance of the Django requests you make.

Creating the Data Models

Our demo application is simple. It only manages books and their relationships.

# models.py
from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=200)

class Publisher(models.Model):
    name = models.CharField(max_length=200)

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)
    publisher = models.ForeignKey(Publisher, on_delete=models.CASCADE)

We need to run the following commands to actually create the database tables:

python manage.py makemigrations
python manage.py migrate

Populating the Database

We need to populate the database to simulate real-world usage.

First, we start the shell:

python manage.py shell

Then, we fill the database with a large amount of data to demonstrate ORM’s shortcomings:

import random

from inventory.models import Author, Book, Publisher

authors = [Author.objects.create(name=f"Author{i}") for i in range(20)]
publishers = [Publisher.objects.create(name=f"Publisher{i}") for i in range(10)]

for i in range(1000):
    Book.objects.create(
        title=f"Book{i}",
        author=random.choice(authors),
        publisher=random.choice(publishers),
    )

Now that we have 1,000 books in our database, it’s easier to spot where ORM is slowing things down.

Creating a Slow Django Endpoint

Let’s set up an endpoint that returns all the books in our database.

from django.http import HttpResponse, JsonResponse

from .models import Book

def home(request):
    return HttpResponse("Hello books app")

def books(request):
    books = Book.objects.all()

    data = []
    for book in books:
        data.append(
            {
                "title": book.title,
                "author": book.author.name,
                "publisher": book.publisher.name,
            }
        )

    return JsonResponse(data, safe=False)

Then add this endpoint in urls.py:

from django.urls import path

from inventory.views import books, home, test_error

urlpatterns = [
    path("", home, name="home"),
    path("books/", books),
]

At first glance, our code looks perfectly normal and doesn't contain any obvious issues. However, it has a performance pitfall that’s common in Django applications.

Inside the loop, it fetches the related objects:

book.author.name
book.publisher.name

However, because the relationships were not prefetched, this will cause extra database queries for each access.

This is known as the N+1 query problem.

Triggering the Slow Request

First, start a development server:

python manage.py runserver

Then send a few requests to the endpoint:

http://127.0.0.1:8000/books/

Each request will generate trace data that AppSignal captures.

Step 1: Finding the Slow Endpoint

Send a few hits to the endpoint. Then, check the AppSignal dashboard, and go to: Performance -> Issue list

The Issue list includes all endpoints used by the application, along with request counts and response times.

These are the results for our demo application:

Endpoint	Mean response time	Requests
GET /books/	4.81 sec	11
GET /home	1.07 sec	7

The /books/ endpoint clearly stands out.

The response time is over 4 seconds, which is considered long for a simple API request.

Step 2: Analyzing Request Performance

AppSignal also provides performance charts to visualize how request latency varies over a period of time.

The charts offer two types of data:

Mean response time: Average request latency
90th percentile latency (p90): Slower request latency that impacts user experience

The spikes on the graph indicate slower request execution times for the /books/ endpoint during testing: almost 6 seconds. This is a confirmation that the /books/ endpoint is performing poorly.

Interestingly, the Slow queries section on AppSignal is empty. This is because the slower request execution times aren’t caused by a single expensive query. Instead, the request is executing hundreds of small queries, which is in accordance with the N+1 query anti-pattern.

Understanding the Root Cause

Django QuerySets are lazy.

When we access a related object, such as book.author, Django makes an additional database query unless the relationship has already been loaded.

So, here’s what’s happening in our loop:

1 query to fetch books
+1000 queries to fetch authors
+1000 queries to fetch publishers

The number of queries is directly proportional to the number of results. This isn’t a problem with the database itself, it’s actually an ORM issue, and request tracing helps us understand the difference between the two.

Fixing the Slow Query

Django offers an optimization feature called select_related() that lets you retrieve related data in the query. This way, it can load all the models upfront, to avoid running thousands of extra queries.

We will update the view:

def books(request):

    books = Book.objects.select_related("author", "publisher")

    data = []
    for book in books:
        data.append(
            {
                "title": book.title,
                "author": book.author.name,
                "publisher": book.publisher.name,
            }
        )

    return JsonResponse(data, safe=False)

The updated query tells Django to join the tables into one query instead of running multiple.

Comparing the Results

As you can see in the table below, after refreshing the endpoint once the fix has been applied, AppSignal shows that the request’s response time is much shorter:

Version	Query strategy	Response time
Original implementation	N+1 queries	3–6 seconds
Optimized implementation	`select_related()`	~60 ms

Here’s one of the traces recorded after the fix has been applied:

GET /books/ → 60 ms

And the response time has been dramatically reduced:

This only confirms that removing the N+1 queries has a significant impact on request times.

A Repeatable Debugging Workflow

Performance debugging is much easier when a standard process is followed.

Here’s how it works with AppSignal:

Identify which endpoints are slow using the Actions dashboard.
Open a representative trace.
Determine what kind of operation is taking up most of the time.
Optimize the code.
Measure the improvement using new traces.

This kind of process represents a much more systematic approach to performance debugging.

Why Observability Matters

Performance issues can creep up on you as the volume of your data increases. Without proper monitoring, issues like N+1 queries can go unnoticed until user experience begins to degrade and noticeable delays occur.

Observability tools like AppSignal offer:

Request-level visibility lets you see what actually happens during a single request, so you’re not guessing where the time is going.
Database vs. app time helps you quickly figure out if the slowdown is coming from queries or your own code.
Performance trends show how things change over time, so you can catch problems early, before they reach users.
Measurable before and after performance comparisons prove that your changes have actually improved performance.

This way, your team can identify bottlenecks and ensure that optimizations are indeed making the system faster.

Conclusion

Slow endpoints don’t usually manifest themselves with obvious error messages. They sneak up on you quietly, stealing time and resources as your app grows.

In our example, AppSignal has helped us debug a slow Django endpoint and identify a classic issue called N+1 queries. With a simple change to our ORM, we cut our request time from several seconds to a few milliseconds.

For a backend developer, request tracing offers a powerful perspective on how your app is behaving. With good observability tools in place, it’s much easier to debug and resolve performance issues.

Tracking Celery Task Failures in Python

Roy Tomeij — 2026-04-09T00:00:00+00:00

Whenever you place an order on Amazon (or any other e-commerce site for that matter), you get that “order placed successfully” notification almost instantly. But did you know that there’s much more to the whole experience than meets the eye?

In Python applications, Celery is the major driver behind the whole thing. The tasks that take time are queued and sent to brokers. Workers pick them up and process them, and the application still runs normally, independent of what is going on in the background.

In this guide, we’ll explain how Celery task failures behave and how you can keep track of everything that goes wrong. We will reproduce a task failure through an example and observe the entries in worker logs. Then, we will use AppSignal to track the failure with full context.

Celery as a Default for Background Jobs in Python

Celery is often in charge of some time-consuming tasks like processing huge datasets or calling external APIs. It plays a huge role in why websites feel so fast. You know how you get an instant success message whenever you place an order or submit a form? Well, there’s a lot of heavy lifting going on behind the scenes.

However, the main problem with Celery is that background tasks fail quietly. From the outside, everything will look healthy because it will manage to succeed after a couple of retries. However, if you take a closer look, you’ll see that multiple errors occurred before this success. In fact, teams generally discover these failures only when a user asks why a certain action didn’t go through.

Celery uses default worker logs to report errors, which is fine when your system is small, but once you’ve got tons of tasks running across multiple workers and queues, it all gets messy fast. Suddenly, engineering teams are left squinting at lines of text, trying to figure out what happened.

The Problem with Celery’s Defaults

Celery’s defaults make background jobs easy to run, but they can also hide reliability issues. A task may fail several times and move through retries before it eventually succeeds, so from the outside everything looks fine while the real errors stay hidden in worker logs. To understand how these defaults can impact your system's stability, let’s look at a practical example of how Celery handles a failing task.

Celery Runs, Fails, and Retries

Celery has built-in retry mechanisms that automatically rerun tasks after an exception. In our application example, we reproduced a task that fails intentionally most of the time:

#A task that fails often
@app.task(bind=True, max_retries=3)
def fragile_task(self):
    attempt = self.request.retries + 1
    retries_left = self.max_retries - self.request.retries

    print(
        f"fragile_task attempt {attempt} "
        f"(task_id={self.request.id}, retries_left={retries_left})"
    )

    time.sleep(1)

    if random.random() < 0.7:
        raise self.retry(exc=Exception("Simulated failure"), countdown=2)

    return "Success"

Our task is designed to fail 70% of the time, with a maximum of three retries before it finally gives up. So when we trigger the task using our producer.py script, Celery will send the job to the message broker (RabbitMQ in our case) and the worker will begin to execute it.

This is our producer.py code:

from tasks import add, fragile_task

# Send a simple task
result1 = add.delay(4, 6)
print("Add task sent:", result1.id)

# Send several fragile tasks
for i in range(5):
    result = fragile_task.delay()
    print("Flaky task sent:", result.id)

At this point, the worker begins processing tasks in the background.

Exceptions Go to Logs Nobody Watches

Celery comes in handy when offloading work to background workers. The flow is simple: you push a task to a queue, the worker picks it up, and your web app stays fast. However, the downside is that Celery failures can be easy to miss.

When a task fails, the traceback just gets printed in worker process logs. So, unless you are actively tailing those logs, there is a good chance they’ll stay under the radar. There’s no central place where you can see failures or track trends over time.

Let's do a test run and look at the logs the worker produces:

Here, we see the worker connect to the RabbitMQ broker (amqp://guest@127.0.0.1:5672) and register two tasks: tasks.add and tasks.fragile_task.

The first one, add, executes immediately and succeeds.

Next, the worker receives several fragile_tasks jobs, which are tasks that we intentionally made unstable with a 70% chance of failing. So when a failure occurs, Celery logs the exception, such as fragile_task attempt 1, and schedules a retry (Retry in 2s: Exception('Simulated failure')). After retrying, most tasks eventually succeed, which is why you later see succeeded in ~1s: 'Success' .

If you look closely at the test run above, you’ll notice the worker logs include entries like this:

[2026-03-04 19:27:10,301: INFO/MainProcess] Task tasks.fragile_task[5cc33856-7038-4d54-9952-f7236c0accf9] received
[2026-03-04 19:27:10,303: INFO/MainProcess] Task tasks.fragile_task[7173fb28-6ea9-4517-a5fc-d8035600fde8] received
[2026-03-04 19:27:10,303: INFO/ForkPoolWorker-2] Task tasks.fragile_task[167b7c9e-7d46-4792-b7ab-d4dd0fbd4e6b] retry: Retry in 2s: Exception('Simulated failure')
[2026-03-04 19:27:10,303: INFO/ForkPoolWorker-3] Task tasks.fragile_task[7173fb28-6ea9-4517-a5fc-d8035600fde8] retry: Retry in 2s: Exception('Simulated failure')
[2026-03-04 19:27:10,303: INFO/ForkPoolWorker-10] Task tasks.fragile_task[5cc33856-7038-4d54-9952-f7236c0accf9] retry: Retry in 2s: Exception('Simulated failure')

From these logs, we can see that the task failed multiple times before reaching its retry limit. Although this information technically exists, it is buried deep in the worker logs. As the system scales and multiple workers run at the same time, the logs quickly become noisy and difficult to interpret.

Retries Hide Real Failures

A task might fail a few times, retry automatically, and eventually succeed. For an untrained eye, this looks like a win. But there are exceptions behind the scenes that you didn’t see. For example, a task that has failed four times and succeeded in its fifth attempt has already raised four exceptions, consumed worker time, and caused a delay for other jobs.

You are unable to see the failures unless you take the time to dig through the worker logs. But this is not practical, especially when you’re running a distributed system with a massive user base that generates loads of log data.

You Find Something Is Broken From Users

Sometimes, a background job fails silently for some users, and retries fail to fix it, as well. However, you only realize this days later, after a user complains about a payment that didn’t go through or a confirmation email that never arrived.

And that’s the issue with Celery and the model it follows. It simply raises an exception and forgets about it, which is good for throughput, but it can cause failures to slip by under the radar.

What AppSignal Captures

In the previous section, we intentionally created failing tasks and triggered them. While the worker logs showed retries and exceptions, we had to manually track task IDs across multiple log lines to understand what actually happened.

This is where AppSignal is useful. It automatically instruments Celery tasks and records every execution for each run. Instead of digging through messy logs, it groups and displays task failures in a structured way, making it easy to see what went wrong and why.

Automatic Task Instrumentation

We’ve already seen how demanding it is to go through worker logs when undertaking investigations. In a distributed system, it’s almost impossible since you are supposed to manually scan thousands of lines of worker logs. AppSignal addresses this issue by removing the need to sift through those.

Once it’s installed, all background tasks are immediately instrumented. It’s capable of gathering structured data regarding each task's execution, including retry attempts, exceptions, and performance metrics. Now, you can see clearly how background jobs are behaving.

Exceptions With Full Tracebacks

By now, AppSignal has been installed, and you’ve set it up to monitor your Python application. You can immediately see how quickly it provides a full trace of task retries.

AppSignal records the complete traceback alongside rich contextual information. You no longer have to dig through logs manually. It shows you exactly what went wrong; you’ve got the exception type, stack trace, when the error occurred, and which worker executed the task. Errors are also grouped by exception type, which makes recurring problems easier to spot.

Retry Visibility

Logs can get pretty messy, even in a simple app, but imagine what it would look like in a distributed system that has thousands of tasks, each generating tons of worker logs and going through multiple retries.

AppSignal makes keeping track of all that a breeze. It shows important details like how many retries happened, when they occurred, and exactly where the error popped up in your code. Without it, a task that fails a few times before finally succeeding might look totally fine on the surface, even though it’s actually failing repeatedly behind the scenes.

Task Context and Arguments

As a software engineer, you already understand the significance of context. When debugging a failure, most of us want to know why something happened and whether it was caused by a new change or something else entirely.

AppSignal helps you out by recording task arguments that led to a failure. This allows you to correlate errors with specific inputs while automatically filtering sensitive information. When you can see both the exception and task context together, debugging becomes much faster.

Installing AppSignal

Enabling Celery monitoring with AppSignal is pretty simple: you only need to install it once in the application environment and initialize it when the worker starts.

The full setup process is covered in the AppSignal Python installation guide, which explains how to:

Install the AppSignal package
Configure your API key
Start the application with monitoring enabled

Seeing Failures as They Happen

In distributed systems, task failures often get buried in worker logs. A task may eventually succeed after a few retries, making the issue seem temporary. But when a task fails repeatedly across multiple workers, the real problem can easily go unnoticed.

AppSignal surfaces these failures immediately, as it captures exceptions directly from the worker process. Instead of hunting through logs across multiple machines, it allows you to view failures as structured events.

In this example, we’ve instrumented the Celery worker with AppSignal and triggered several failing fragile_task executions to demonstrate how it all works.

Celery Task Failures Appear in the Errors Dashboard

When the failing tasks run, AppSignal catches the exceptions and bundles identical failures together. Instead of spamming you with a log entry for every retry, it aggregates them into a single issue on the dashboard.

This makes it easy to spot a recurring problem, as you can see in this AppSignal dashboard:

This dashboard interface shows:

Failing background task, run/tasks.fragile_task
Exception grouped under one issue
Six total occurrences
The time the error last occurred

This grouping is a game-changer in distributed workloads. A single bug can produce dozens (or even hundreds of retries) across workers. AppSignal spots these patterns right away, so you don’t have to dig through logs to see what’s going on.

Inspecting the Error Details

Click the grouped error to get a detailed view of the failure. You’ll find critical debugging information (normally scattered across worker logs) grouped together.

Here’s what the dashboard shows:

You will find:

Error message: This is the exception raised during task execution; in this example, the task fails with Simulated failure message.
Full traceback: The stack trace shows exactly where the error occurred in the code. Here, the failure originates from tasks.py inside the fragile_task.
Task metadata: Each task execution includes a message ID generated by the queue system.
Execution environment: The worker hostname (Nehemiah) identifies which worker processed the task.
Queue information: The task was executed from the celery queue.
Timestamp: The exact time the failure occurred is recorded for every sample.

All this helps you see whether failures were isolated events or if they actually happened repeatedly across multiple workers. This context is very important in distributed task systems. Plus, as mentioned, you don’t have to dig for this information yourself. You get a clear view of which task failed, where it ran, and how often this failure occurred.

These attributes provide deeper operational context such as:

retry count (celery.retries)
worker hostname (celery.hostname)
queue routing information
task state (FAILURE)

Debugging no longer requires you to work on something vague that failed in the queue. It has become a specific Celery task that failed with this exception across multiple executions.

Why This Matters in Production Systems

Failures are pretty easy to spot in small development environments, but what if the production system runs dozens of Celery workers with multiple processes each?

For example, the worker in this demo runs with 12 concurrency, as seen here:

This means up to twelve tasks execute simultaneously. When failures occur under these conditions, logs can grow quickly and conceal the underlying issue.

AppSignal solves this problem by:

Capturing exceptions automatically
Grouping failures by type
Attaching execution metadata
Showing trends in the error dashboard

Instead of reading logs line by line, you can immediately identify which task is failing and why.

Patterns to Watch For

Observability is more than just capturing individual failures. As an engineer, you will often encounter patterns that reveal deeper issues in background job systems. Observability tools monitoring dashboards, just like we have seen with AppSignal, make these patterns visible before they become huge production problems.

Retry Storms

A retry helps a task recover from temporary failures, but the downside is they can hide deeper problems. We can equate retries to safety nets of distributed systems designed to handle temporary failures. It is great for improving a system's resilience but can also mask underlying problems.

Often a task retrying several times before succeeding generates multiple failures that might go unnoticed. If you are not monitoring the retry count alongside the success rates, you might miss the fact that tasks are succeeding only after multiple attempts.

The same exception might appear multiple times and then succeed. The retry is not fixing the problem but rather delaying failure. If you see the same exception recurring across multiple retries, then there is a high likelihood you are dealing with a deeper issue that will disrupt the system rather than a random error.

One Task Dragging Down the Queue

Background tasks have distinct behaviors. Some tasks are processed almost immediately, while others take a few seconds. When a slow task occupies a worker process for too long, it reduces the number of workers available for other tasks. This results in slow queue latency, delayed task execution, and reduced system throughput.

External Dependency Failures

There are background tasks that have to interact with some external systems, like APIs, databases, or messaging services. These tasks can appear broken when those dependencies fail, even though they are not the real problem.

Tracebacks can reveal these issues. Some common examples include ConnectionError, TimeoutError, HTTPError. In these cases, the task logic may be correct, but the external service it depends on could be unavailable or the response may be slow.

Alerting on What Matters

Dashboards help when investigating issues, but alerts ensure problems are detected immediately. In background processing systems like Celery, tasks often run silently without direct user interaction. This means failures can continue for hours before anyone notices unless alerting is configured properly.

Effective alerting involves targeting signals that indicate deeper system issues rather than temporary noise. It’s important to always target high-value signals, as they help find the root cause of a problem.

High-value alerts typically include:

Error Rate Spikes

If a task that normally succeeds suddenly begins to fail frequently, it signals a bug, a configuration issue, or a missing dependency. Alerting on unusual error rates helps teams react quickly before failures affect larger workflows.

You can easily spot the tasks that are slowing down the system and fix the issue by going through the data.

New Exception Types

When you spot new exceptions that you haven’t noticed before, new system changes might be at fault. If engineers manage to detect them early enough, they will be able to investigate and fix everything before the issue spreads.

Task Duration Anomalies

A task that normally completes in milliseconds but suddenly starts taking seconds can slow down queue processing. Alerting on performance anomalies helps identify slow queries and overloaded services.

Engineers’ goal is to know when something important breaks. For example, if a task such as send_welcome_email starts failing repeatedly, the team should receive an alert immediately rather than by discovering the issue through user complaints later.

Well-designed alerts focus on high-value signals that help indicate deeper system issues.

Example: Debugging a Silent Failure

Now, let’s consider a background task called process_payment. Its job is to charge a user’s card and update their account once the payment succeeds. Most payments complete successfully, but suddenly there is a small percentage that begins to fail. Since the task is configured to retry, some payments eventually succeed after multiple attempts. From the outside, the system appears healthy.

Without proper visibility, the engineering team will only become aware of this problem when a user submits a support ticket. They will then search through worker logs trying to reconstruct what happened, often without enough context to reproduce the failure.

AppSignal makes investigation easier. The error dashboard quickly reveals a spike in stripe.error.CardError exceptions coming from the process_payment task. Instead of scattered log entries, failures are grouped together under a clean interface.

Looking at the error samples shows additional context:

User IDs associated with the failures
Timestamps of each occurrence
Traceback returned by the Stripe API
Number of retries that occur

From there, the pattern becomes clear. Failures occur when a specific card type fails Stripe’s validation rules. Once the pattern is visible, fixing the issue becomes easier.

Wrap-Up

Celery enables workers to run in the background, which represents a big advantage for modern software systems. A task is sent to a broker (RabbitMQ or Redis) and then assigned to workers for processing. Tasks throwing exceptions are retried without affecting the functioning of the whole application. However, these retries can also make it difficult to notice underlying issues that eventually trigger system failures.

In most cases, background tasks fail quietly, especially with no proper visibility. In huge distributed systems, exceptions disappear into worker logs. Over time, it becomes a challenge for teams to discover problems before users report them.

Observability platforms help in addressing this challenge. Instrumenting Celery tasks enables engineering teams to see failures as they happen, track retries, and inspect the full context behind each exception.

If you’re already running Celery in production, the next step is straightforward. Follow the Python installation guide in the AppSignal documentation to add monitoring to your application and start capturing real task behavior.

Improve Query Performance Using Python Django QuerySets

Roy Tomeij — 2025-12-03T00:00:00+00:00

When developing web applications with Django, your interaction with the database impacts overall application performance. Django's Object-Relational Mapper (ORM) is a powerful ally that offers an intuitive way to work with your data through abstractions called QuerySets. These are your primary tools for fetching, filtering, creating, and managing data.

In this article, we'll explore fundamental — yet highly effective — techniques to optimize your Django QuerySets. Our goal is to learn how to write database queries that are both functional and efficient. This ensures your applications remain fast, responsive, and scalable.

First, let's dig deeper into why database performance shouldn't be viewed as just a technical concern, but rather a cornerstone of a successful web application.

The Importance of Database Performance in Web Applications

A database is often the workhorse of a web application. It serves and stores the data that brings your application to life. Its performance is not an isolated metric. It has far-reaching consequences for both the end-user experience and the operational health of your servers.

So, let’s briefly discuss how performance impacts user experience and server resources in an application.

Impact on User Experience

Slow database queries are a primary culprit behind high page load times. Users’ patience is a finite resource, and when an application feels sluggish, they are more likely to leave. Over time, poor performance can lead to lower engagement and create a negative perception of your brand, eroding user trust and portraying your application as unreliable. Optimizing database performance, therefore, is directly tied to user satisfaction and retention.

Impact on Server Resources

Inefficient queries also place a strain on your server infrastructure. By forcing the database to perform complex joins or scan large tables, they consume excessive hardware resources. This pressure also limits your application's ability to scale. In extreme cases, an overloaded database can become unresponsive, leading to cascading failures that risk an application outage. Writing efficient QuerySets ensures your application makes judicious use of server resources. This leads to a stable, scalable, and cost-effective system.

Read our guide Monitor the Performance of Your Python Django App with AppSignal for more on this topic.

Understanding Django QuerySets

Before going through specific optimization techniques, it is important to have an overview of what QuerySets are, their core characteristics, and why they are a cornerstone of Django's database interaction model.

What is a QuerySet?

A Django QuerySet is a list of objects from your database. It is an abstraction layer that represents a collection of database queries. It allows you to retrieve, filter, order, and annotate data from your database using Python, rather than writing raw SQL. When you interact with your Django models, you are working with QuerySets.

Think of a QuerySet as a "recipe" for fetching data. You can define this recipe step-by-step, and Django will only execute when it absolutely has to.

Key Characteristic of QuerySets: Laziness

One of the most important characteristics of QuerySets is their "laziness." This means that the act of creating or modifying a QuerySet does not immediately result in any database activity. For example, consider the following:

from myapp.models import User
# No database query has been executed yet
active_users = User.objects.filter(is_active=True)
recent_signups = active_users.filter(date_joined__year=2025)
ordered_users = recent_signups.order_by('-date_joined')

In this code, even though you have defined active_users, recent_signups, and ordered_users, Django has not sent a SQL query to the database. Instead, Django simply builds an internal representation of the query.

The database query is executed when the QuerySet is evaluated. Evaluation typically occurs when you:

Iterate over the QuerySet: This is perhaps the most common scenario for evaluation. When you start a loop (like a for loop) over a QuerySet, Python needs the actual items to iterate through. At this point, Django executes the database query to fetch the rows that match your QuerySet's criteria. The database returns the data, and Django begins to instantiate model objects one by one (or in internal chunks for efficiency) as your loop progresses. The key here is that the loop's body needs the data to operate on each item.
Slice it with a step parameter: Basic slicing (like ordered_users[:5] to get the first five users) often returns another unevaluated QuerySet. Instead, if you use a step in the slice (e.g., [::2] to get every second user), you force immediate evaluation. To determine which items to return when a step is involved, Django typically needs to retrieve the entire result set from the database first. Once all potential items are fetched into memory, Python can apply the stepping logic to select the appropriate items. This is different from simple limit/offset slicing, which the database can often handle directly.
Pickle or cache it: Pickling converts a Python object into a byte stream, often for storage or transmission. If you attempt to pickle a QuerySet, you generally want to pickle its results, not just the abstract query definition. So, to serialize the data, you (or the system) must first evaluate the QuerySet to fetch that data from the database. Similarly, when you cache a QuerySet, you usually aim to store its results to avoid future database hits. This process needs an initial database query to retrieve the results that you (or the system) will then place into the cache.
Call repr() or len() on it:
- Calling repr() on a QuerySet will trigger its evaluation. To provide a string representation, Django usually needs to fetch at least a subset of the results to display (e.g., <QuerySet [<User: Alice>, <User: Bob>, ...]>).
- Calling len() on a QuerySet also causes evaluation. To determine the number of items in the QuerySet, Django will execute the query, retrieve all the matching objects into memory, and then count how many there are. This is why it's generally less efficient than using the queryset.count() method if all you need is the count. In fact, it performs a more optimized SELECT COUNT(*) at the database level.
Explicitly convert it to a list: When you explicitly try to convert a QuerySet into a Python list using the list() constructor, you are signaling that you need all the results of that query available in memory as a standard Python list. This forces Django to execute the database query, retrieve all matching rows, instantiate them as model instances (or dictionaries/tuples), and populate the list with these items.
Call methods that return a single value or object: Many QuerySet methods are designed to return a specific piece of information or a single object, rather than a collection.
- count(): To return the total number of matching records, the database must be queried to perform the count.
- exists(): To determine if at least one matching record exists, a query must be sent to the database (though it's highly optimized to stop at the first find).
- first(), last(), earliest(), latest(): These methods are designed to retrieve a single specific object from the ordered QuerySet. This requires a database query to find and fetch that particular record.
- get(): This method is used to retrieve a single, unique object matching the given lookup parameters. It inherently requires a database query. If no object or multiple objects are found, it raises an exception, which also implies the query was executed.
- Methods like aggregate(), earliest(), latest() also trigger immediate database queries because they compute a result based on the entire QuerySet.

This lazy evaluation is beneficial on the side of database performance because:

It allows you to chain multiple filters and operations efficiently. Django can optimize the entire chain into a single, more efficient SQL query.
You can create and pass QuerySets around in your code without incurring unnecessary database hits until the data is actually needed.

Why Use Django QuerySets?

QuerySets are a fundamental part of Django's ORM for several reasons:

Pythonic interface: They provide a clean, intuitive, and Pythonic way to interact with your database. This makes your data access code readable, maintainable, and easier to understand.
Abstraction over SQL: QuerySets abstract away the complexities and variations of SQL syntax across different database backends. Django handles the translation from your Python QuerySet methods to the appropriate SQL for your configured database. This also provides a significant degree of database portability.
Security: By default, QuerySets protect against SQL injection attacks because parameters are escaped correctly by the database driver. Writing raw SQL queries manually can be error-prone and open up security vulnerabilities if not handled with extreme care.
Developer productivity: They significantly speed up development by reducing the amount of boilerplate code needed for common database operations.
Built-in features for optimization: As you are about to see, Django's ORM and QuerySets come with several built-in features and methods designed to help you write more performant database queries.

Implementing Basic QuerySet Performance Techniques

Let's implement some basic, yet impactful, performance techniques for Django QuerySets.

Requirements, Dependencies, and Settings

To reproduce this tutorial, you need to have Python 3.6 or newer installed.

Suppose you call the main folder of your project django_query_sets/. At the end of this step, the folder will have the following structure:

django_query_sets/
     └── venv/

Where venv/ contains the virtual environment. You can create the venv/ virtual environment directory like so:

python -m venv venv

To activate it, run this on Windows:

venv\Scripts\activate

On macOS and Linux, execute:

source venv/bin/activate

In the activated virtual environment, install the dependencies with:

pip install django

This will install the latest version of Django.

Create your Django project as query_sets_project/ in the django_query_sets/ folder type:

django-admin startproject query_sets_project

Your folder now has the following structure:

django_query_sets/
     ├── query_sets_project
     │     ├── query_sets_project
     │     │     ├── __init__.py
     │     │     ├── settings.py
     │     │     ├── urls.py
     │     │     ├── asgi.py
     │     │     └── wsgi.py
     │     └── manage.py
     │
     └── venv/

The query_sets_project/ folder contains the main project files. In this tutorial, you will create an application that you can call catalog. To do so, from django_query_sets/ navigate to query_sets_project/:

cd query_sets_project

Now, you can instantiate the catalog/ folder:

python manage.py startapp catalog

The whole project now has the following structure:

django_query_sets/
    ├── query_sets_project/
    │        ├── manage.py
    │        ├── query_sets_project/
    │        │      ├── __init__.py
    │        │      ├── asgi.py
    │        │      ├── settings.py
    │        │      ├── urls.py
    │        │      └── wsgi.py
    │        └── catalog/
    │               ├── __init__.py
    │               ├── admin.py
    │               ├── apps.py
    │               ├── models.py
    │               ├── tests.py
    │               ├── views.py
    │               └── migrations/
    │                     └── __init__.py
    └── venv/

Go to the django_query_sets/query_sets_project/query_sets_project/settings.py file. Inside it, there is a list of installed apps which appears as follows:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]

You need to add your catalog application to the list:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    'catalog.apps.CatalogConfig', # Add catalog app to the list
]

Perfect! You are now ready to start writing Django code and using QuerySets!

Setting up a Sample Model

Let's define a couple of Django models in catalog/models.py.

Create Author and Book models, establishing a one-to-many relationship:

from django.db import models
from django.utils import timezone

class Author(models.Model):
    name = models.CharField(max_length=100, unique=True)
    bio = models.TextField(blank=True, null=True) # Author biography

    def __str__(self):
        return self.name

class Book(models.Model):
    title = models.CharField(max_length=200)
    # related_name='books' allows us to do author_instance.books.all()
    author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name='books')
    publication_date = models.DateField()
    pages = models.IntegerField()
    price = models.DecimalField(max_digits=6, decimal_places=2)
    # ISBN is unique, but can be nullable if not always available
    isbn = models.CharField(max_length=13, unique=True, null=True, blank=True)

    class Meta:
        ordering = ['title'] # Default ordering for Book queries

    def __str__(self):
        return self.title

After defining or modifying your models, you must create and apply database migrations (via the CLI):

python manage.py makemigrations catalog
python manage.py migrate

Below is the result:

This updates your database schema to reflect your model definitions.

Now, populate the database with some sample data. You can do this via the Django admin interface (after registering your models in admin.py) or programmatically, using the Django shell (python manage.py shell).

In this case, we will use the Python shell. Inside the django_query_sets/query_sets_project/ folder, type:

python manage.py shell

Paste the following code into the CLI:

from catalog.models import Author, Book
from django.utils import timezone
import datetime

# Create an author
author1 = Author.objects.create(name="George Orwell")
author2 = Author.objects.create(name="J.R.R. Tolkien")

# Create some books
Book.objects.create(
    title="1984",
    author=author1,
    publication_date=datetime.date(1949, 6, 8),
    pages=328,
    price=15.99,
    isbn="9780451524935"
)
Book.objects.create(
    title="Animal Farm",
    author=author1,
    publication_date=datetime.date(1945, 8, 17),
    pages=112,
    price=12.50,
    isbn="9780451526342"
)
Book.objects.create(
    title="The Hobbit",
    author=author2,
    publication_date=datetime.date(1937, 9, 21),
    pages=310,
    price=14.00,
    isbn="9780547928227"
)
Book.objects.create(
    title="The Lord of the Rings",
    author=author2,
    publication_date=datetime.date(1954, 7, 29),
    pages=1178,
    price=25.00,
    isbn="9780618640157"
)

print("Sample data created successfully!")

When done, type exit().

The data you have created will now be persisted in your database. You are ready to query it using QuerySets!

Fetching Specific Fields with `values()` and `values_list()`

By default, when you retrieve objects from a database, Django fetches all fields for each object. If you only need a subset of these fields, fetching everything is wasteful. This is where values() and values_list() come in handy:

values(*fields): Returns a QuerySet that returns dictionaries, rather than model instances, when used as an iterable. Each dictionary represents an object, with the keys corresponding to the field names you specified.
values_list(*fields, flat=False): Similar to values(), but it returns tuples instead of dictionaries. If you only specify one field, you can use flat=True to get a flat list of single values.

This is more performant because fetching only specific fields reduces the amount of data transferred from the database to your application and decreases the memory required to hold the results. It also avoids the overhead of creating full model instances if you do not need them.

For example, say you only need the titles and publication dates of all books. In catalog/views.py, write the following:

from django.shortcuts import render
from django.http import HttpResponse
from .models import Book

def book_titles_and_dates_view(request):
    # Using values()
    book_data_dicts = Book.objects.values('title', 'publication_date')

    output_lines = ["<h3>Book Titles and Publication Dates (using values()):</h3>"]
    for book_dict in book_data_dicts:
        # Accessing data by dictionary keys
        output_lines.append(f"Title: {book_dict['title']}, Published: {book_dict['publication_date']}")
    return HttpResponse("<br>".join(output_lines))

def book_titles_only_view(request):
    # Using values_list()
    book_titles = Book.objects.values_list('title', flat=True)

    output_lines = ["<h3>Book Titles Only (using values_list(flat=True)):</h3>"]
    # Iterating over a flat list of titles
    for title in book_titles:
        output_lines.append(title)

    # Print query to console
    print("--- Book Titles (from values_list with flat=True) ---")
    for title in book_titles:
        print(title)
    print("----------------------------------------------------")
    print(str(book_titles.query))

    return HttpResponse("<br>".join(output_lines))

To see these views in action, you need to map them to URLs. Create a catalog/urls.py file and write the following:

from django.urls import path
from . import views

app_name = 'catalog' # Optional but good practice for namespacing

urlpatterns = [
    path('titles-dates/', views.book_titles_and_dates_view, name='book_titles_dates'),
    path('titles-only/', views.book_titles_only_view, name='book_titles_only'),
]

Then, modify query_sets_project/query_sets_project/urls.py to include the catalog/urls.py file. The code you will find in query_sets_project/query_sets_project/urls.py is:

from django.contrib import admin
from django.urls import path

urlpatterns = [
    path("admin/", admin.site.urls),
]

Change it as follows:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('catalog/', include('catalog.urls')),
]

To see the results, navigate to your project's root directory (django_query_sets/query_sets_project/) in your terminal and run:

python manage.py runserver

To see the results, go to the dedicated URLs:

When the titles-only views run, you can see the resulting query via the CLI:

Now, to see the difference in performance with the "standard method", you can add this method in catalog/views.py:

def all_book_details_view(request):
    # This fetches ALL fields for every book from the database.
    all_books = Book.objects.all()

    # Print SQL query
    print(str(all_books.query))

    output_lines = ["<h3>All Book Details (Default Method):</h3>"]
    for book in all_books:
        output_lines.append(book.title)

    return HttpResponse("<br>".join(output_lines))

The database is doing the work of retrieving every column, and Django is building a full model instance for each row, only for us to use a single field: title. This is the inefficiency eliminated by values() and values_list().

In catalog/urls.py, add a dedicated URL:

urlpatterns = [
    # Existing code not shown for brevity
    path('all-details/', views.all_book_details_view, name='all_book_details'), # Add this line
]

Rerun the server and go to http://127.0.0.1:8000/catalog/all-details/. The result you see in the UI is the same as at http://127.0.0.1:8000/catalog/titles-only/. What changes is the query under the hood. You can see it printed in the command line:

This shows that you are obtaining the same result (listed titles), but with a longer and inefficient query.

Efficient Counting with `count()`

Often, you just need to know how many objects match a particular query, not the objects themselves. A naive approach might be to fetch all objects and then use len() on the resulting list:

all_books = Book.objects.all()
number_of_books = len(all_books) # This evaluates the QuerySet and loads all book objects

This is inefficient because it loads all the book objects into memory just to count them. Django QuerySets provides a much better way: the count() method.

It performs a SELECT COUNT(*) (or a similar count-specific query) at the database level. This is significantly faster and uses far less memory than retrieving all objects.

Let's add a view to demonstrate this. Add the following as a new function in catalog/views.py (do not cancel the others):

from .models import Author # Add this import on top of the file
from django.db import connection # Add this import on top of the file

def book_count_view(request):
    # Total count query
    total_books = Book.objects.count()

    print("\n--- Actual query for total book count ---")
    print(connection.queries[0]['sql'])
    print("-----------------------------------------")

    # Filtered count query
    connection.queries.clear()
    orwell_books_count = Book.objects.filter(author__name="George Orwell").count()

    print("\n--- Actual query for filtered book count ---")
    print(connection.queries[0]['sql'])
    print("--------------------------------------------")

    output_lines = [
        "<h3>Book Counts (using count()):</h3>",
        f"Total books in the library: {total_books}",
        f"Number of books by George Orwell: {orwell_books_count}"
    ]
    return HttpResponse("<br>".join(output_lines))

Now add a dedicated URL in catalog/urls.py:

from django.urls import path
from . import views

app_name = 'catalog'

urlpatterns = [
    # Existing code not shown for brevity
    path('book-counts/', views.book_count_view, name='book_counts'), # Add this line
]

The result is shown at http://127.0.0.1:8000/catalog/book-counts/, after re-running the server:

The console shows the underlying query:

As before, you can create a function that performs the same query (but inefficiently) in views.py:

def book_count_slow_view(request):
    # Fetch all book objects from the database
    all_books = Book.objects.all()

    print("\n--- Query for book_count_slow_view ---")
    print(str(all_books.query))
    print("--------------------------------------")

    # Use len() for loading all objects into memory
    number_of_books = len(all_books)

    output_lines = [
        "<h3>Book Count (Slow Method):</h3>",
        f"Total books in the library: {number_of_books}"
    ]
    return HttpResponse("<br>".join(output_lines))

Add the URL to urls.py:

urlpatterns = [
    # Existing code not shown for brevity
    path('book-counts-slow/', views.book_count_slow_view, name='book_counts_slow'), # Add this line
]

Rerun the server and go to http://127.0.0.1:8000/catalog/book-counts-slow/. You will see the same result in the UI as the one in http://127.0.0.1:8000/catalog/book-counts/. Again, what changes is the query under the hood. You can see it printed in the command line:

The image shows how inefficient this query is, as it retrieves all the data from the database.

Efficient Existence Checks with `exists()`

Sometimes, you only need to know whether at least one object matches your query. For example: "Are there any books by J.R.R. Tolkien in the database?"

To do so efficiently, create a view by adding a new function in catalog/views.py that uses the method exists():

def check_books_exist_view(request):
    # Use the efficient method exists()
    connection.queries.clear()
    tolkien_books_exist = Book.objects.filter(author__name="J.R.R. Tolkien").exists()

    print("\n--- Actual query for exists() check ---")
    print(connection.queries[0]['sql'])
    print("---------------------------------------")

    output_lines = ["<h3>Book Existence Checks (using exists()):</h3>"]
    if tolkien_books_exist:
        output_lines.append("Yes, there are books by J.R.R. Tolkien in the database.")
    else:
        output_lines.append("No books by J.R.R. Tolkien found.")

    return HttpResponse("<br>".join(output_lines))

As before, add a new URL in catalog/urls.py:

urlpatterns = [
    # Existing code not shown for brevity
    path('check-existence/', views.check_books_exist_view, name='check_book_existence'), # Add this line
]

After re-running the server, the result is visible at http://127.0.0.1:8000/catalog/check-existence/ :

The console shows the underlying query:

To obtain the same result, but inefficiently, you could create such a function in views.py:

def check_books_exist_slow_view(request):
    output_lines = ["<h3>Book Existence Checks (Slow Methods):</h3>"]

    # Inefficient method 1: Using count() > 0
    connection.queries.clear()
    # This asks the database to find and count ALL matching books.
    if Book.objects.filter(author__name="J.R.R. Tolkien").count() > 0:
        output_lines.append("Using count(): Yes, books by J.R.R. Tolkien exist.")

    print("\n--- Actual query for count() existence check ---")
    print(connection.queries[0]['sql'])
    print("------------------------------------------------")

    # Inefficient method 2: Evaluating the QuerySet directly
    connection.queries.clear()
    # This fetches ALL matching books into memory.
    if Book.objects.filter(author__name="J.R.R. Tolkien"):
        output_lines.append("Using boolean check: Yes, books by J.R.R. Tolkien exist.")

    print("\n--- Actual query for boolean existence check ---")
    print(connection.queries[0]['sql'])
    print("------------------------------------------------")

    return HttpResponse("<br>".join(output_lines))

Create a new URL in urls.py:

urlpatterns = [
     # Existing code not shown for brevity
    path('check-existence-slow/', views.check_books_exist_slow_view, name='check_book_existence_slow') # Add this line
]

When checking http://127.0.0.1:8000/catalog/check-existence-slow/, you will see this result:

Again, what changes is the query:

In this case, one query counts to the end, and the other fetches too much data. Instead, when using exists(), the optimization of SQL contains the LIMIT 1 clause, which is the database's instruction to perform the absolute minimum work required.

Side-note: While Django's ORM is powerful, it's important to be aware of potential pitfalls like the N+1 query problem, which can significantly degrade performance when fetching related objects. For a detailed guide on identifying and fixing these issues with AppSignal, check out Find and Fix N+1 Queries in Django Using AppSignal.

Wrapping Up

In this article, we explored key techniques for optimizing Django QuerySets. These methods reduce data transfer, minimize memory usage, and lessen the load on your database server, leading to a faster, more responsive, and scalable web application. Mastering these is a crucial step for any Django developer aiming to build high-performance systems.

Happy coding!

Switching from Pip to uv in Python: A Comprehensive Guide

Roy Tomeij — 2025-09-24T00:00:00+00:00

Python has long relied on pip as its standard package manager, but a blazing-fast alternative is now changing the landscape.

uv is a Rust-based package manager that aims to transform Python dependency management with unmatched performance and simplicity.

In this guide, you will learn everything you need to know about uv and how to smoothly transition from the traditional pip ecosystem.

Let's get started!

What is uv?

uv is a high-performance Python package and project manager written in Rust. It was developed by Astral (the company behind the popular Ruff linter), and is designed to be the comprehensive solution for Python package management.

With uv, you can replace a wide range of traditional tools in your workflow, such as:

pip for installing packages.
pip-tools for dependency locking.
virtualenv/venv for environment management.
pyenv for managing Python versions.
pipx for installing CLI tools.
poetry for project configuration and publishing.
twine for distributing packages.

Thanks to its Rust-based roots, uv performs operations 10-100 times faster than these traditional Python tools, and this increase in speed is noticeable even in small projects.

uv also embraces modern packaging standards. It uses pyproject.toml for configuration and introduces a reliable lock file that ensures consistent environments across systems.

Before we explore uv in detail, let's quickly recap the current Python landscape for package management.

Traditional Python Package Approach: `pip` and `virtualenv`

The traditional Python package management workflow relies on a fragmented set of tools, each handling a specific piece of the puzzle.

It typically begins with installing Python, either via system package managers or by downloading it from the official website.

From there, you might use tools like venv or virtualenv to create isolated environments, followed by pip to install packages. Dependency tracking is often manual or handled through add-ons like pip-tools.

A typical workflow with pip and virtualenv might look like this:

# Create a virtual environment
python -m venv .venv

# Activate the environment
source .venv/bin/activate

# Install packages
pip install django requests

# Record dependencies
pip freeze > requirements.txt

While this workflow is well-known and widely used, it involves multiple steps and tools, each with its own syntax and quirks. You must remember to activate environments, manage dependencies manually, and juggle a variety of commands.

This is precisely the pain point uv aims to solve by streamlining the entire process into a faster and more cohesive experience.

Installing uv

Getting started with uv is quick and straightforward, with support across all major operating systems. You can follow the official uv installation guide, or use one of the commands below.

On Linux and macOS, run:

curl -LsSf https://astral.sh/uv/install.sh | sh

On Windows, use PowerShell:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

After installation, confirm that it's working by checking the version:

uv --version

uv 0.7.3

To update uv later, simply run:

uv self update

This will fetch and install the latest version automatically:

info: Checking for updates...
success: Upgraded uv from v0.6.13 to v0.7.3! https://github.com/astral-sh/uv/releases/tag/0.7.3

The easiest way to begin using uv is to replace the tools you already rely on. Since uv is designed as a drop-in replacement, you can take advantage of its performance and simplicity without overhauling your current workflow.

Let's look at how that works in practice.

Managing Virtual Environments with uv for Python

uv can serve as a direct replacement for tools like virtualenv and venv to create isolated Python environments.

Previously, you might run:

python -m venv .venv

With uv, the equivalent is:

uv venv # uses .venv by default

By default, this creates a virtual environment in .venv, but you can specify a custom path if needed:

uv venv <path/to/environment>

After running the command, uv provides a clear summary showing which Python interpreter was used, where the environment was created, and how to activate it in your shell:

Using CPython 3.13.2 interpreter at: /home/ayo/.local/share/mise/installs/python/3.13.2/bin/python
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate.fish

You can also require that a specific version of Python is used:

uv venv --python 3.11

If the version isn't already installed, uv will fetch and install it for you automatically.

In practice, uv's environment creation is incredibly fast. In my tests, it was roughly 200x faster than venv:

Replacing Pip with uv for Dependency Management

uv provides a pip-compatible interface for managing your project dependencies. You only need to prefix specific pip commands with uv. For example, you can install dependencies with:

uv pip install flask requests

There's no need to activate a virtual environment beforehand, as uv automatically detects and uses the nearest environment in your current or parent directory.

If no environment is found, it will prompt you to create one or suggest using the --system flag to install it globally:

Installing dependencies from existing project files works the same way:

uv pip install -r requirements.txt

uv pip install -r pyproject.toml

If you're using pip-tools to compile requirements.txt from a requirements.in file, uv provides a much faster alternative:

pip-compile requirements.in -o requirements.txt

With the uv equivalent:

uv pip compile requirements.in -o requirements.txt

I experienced a ~150x speedup over pip-compile:

Removing dependencies is also straightforward with:

uv pip uninstall flask

Managing Python Versions with uv

One of uv's key advantages is that it doesn't rely on Python to run. Instead, it automatically detects existing Python installations and uses them for any Python-specific tasks.

If a project requires a version of Python that isn't already installed, uv will also download and manage it for you.

You can install Python versions directly like this:

uv python install # the latest version
uv python install 3.13 # a specific version
uv python install 3.13 3.12 # multiple versions

To check which versions are available on your system, run:

uv python list --only-installed

This lists all Python versions installed by uv, as well as any others already on your system, regardless of how they were installed:

cpython-3.13.2-linux-x86_64-gnu     /home/ayo/.local/share/mise/installs/python/3.13.2/bin/python3.13
cpython-3.13.2-linux-x86_64-gnu     /home/ayo/.local/share/mise/installs/python/3.13.2/bin/python3 -> python3.13
cpython-3.13.2-linux-x86_64-gnu     /home/ayo/.local/share/mise/installs/python/3.13.2/bin/python -> python3.13
cpython-3.12.10-linux-x86_64-gnu    /home/ayo/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/bin/python3.12
cpython-3.12.3-linux-x86_64-gnu     /usr/bin/python3.12
cpython-3.12.3-linux-x86_64-gnu     /usr/bin/python3 -> python3.12
cpython-3.11.12-linux-x86_64-gnu    /home/ayo/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/bin/python3.11

With uv, you don't need to install Python ahead of time. If a command requires a version you don't yet have, uv will fetch and install it automatically before proceeding. This on-demand behavior eliminates the need for separate tools like pyenv.

Simplifying Tasks with uv

So far, I've focused on how uv can serve as a faster, drop-in replacement for tools you're already using, allowing you to gain performance benefits with minimal disruption.

But uv also introduces an alternative, more modern workflow that simplifies many tasks and goes beyond what traditional tools offer.

In the next sections, we'll explore some of these new capabilities.

Creating and Managing Python Projects with uv

One of uv's core goals is to manage the entire lifecycle of a Python project, from creating the project itself, to managing its dependencies, running commands or scripts, and even preparing it for distribution.

To start a new project, run:

uv init <project>

By default, this sets up an application-style project. To create a library project instead, use:

uv init --lib <project>

An initialized application project has the following structure:

.
├── .git
├── .gitignore
├── main.py
├── pyproject.toml
├── .python-version
└── README.md

uv sets up version control with Git, includes pyproject.toml for project metadata and dependencies, and creates a basic script (main.py) along with a .python-version file. It's ready to run out of the box.

Execute the script with:

uv run main.py

The first time you run this, uv will select the appropriate Python version, create a virtual environment, and then execute the script:

Using CPython 3.12.10
Creating virtual environment at: .venv
Hello from example-project

At this point, uv generates a uv.lock file, which records the full, resolved dependency graph — ensuring consistent environments across different machines.

While pyproject.toml defines intended dependencies (like requirements.in), uv.lock captures the exact installed versions, effectively replacing requirements.txt for reproducibility.

Managing Dependencies to a uv Project

It's simple to add dependencies to a uv project. To add a package, use:

uv add requests

For development dependencies or tools, append the --dev flag:

uv add --dev black

These changes are reflected in your pyproject.toml:

# pyproject.toml
. . .
dependencies = [
    "requests>=2.32.3",
]

[dependency-groups]
dev = [
    "black>=25.1.0",
]

You can create custom dependency groups using the --group flag:

uv add --group tools black
uv add --group production gunicorn
uv add --group dev pytest # same as uv add --dev

Resulting in:

[dependency-groups]
dev = [
    "pytest>=8.3.5",
]
production = [
    "gunicorn>=23.0.0",
]
tools = [
    "black>=25.1.0",
]

This structure makes it easy to install only the dependencies needed for a given environment. For example, to install just the base and production dependencies, run:

uv sync --no-group dev --no-group tools --group prod

Dependencies are always installed into the project's virtual environment, so there's no need to activate it manually. Once installed, you can run any tool using:

uv run <tool>

As in:

uv run pytest

This finds the tool version in your project's virtual environment and executes it:

uv also provides a tools interface for working with tools that aren't tied to a specific project:

uv tool run <tool> # Same as uvx <tool>

In this case, the tool and its dependencies are installed in a temporary virtual environment and executed accordingly. If you use the tool regularly, you can install it for easier access:

uv tool install <tool>

Upgrading and Removing Dependencies

uv provides straightforward commands for updating and removing packages as your project evolves. Here's how to upgrade a specific package to its latest version:

uv add --upgrade requests

This command updates the package and refreshes the lock file to ensure consistent environments across your team.

Removing packages is equally simple:

uv remove requests

uv will clean up the pyproject.toml, remove unneeded transitive dependencies, and regenerate the uv.lock file automatically.

Migration Guide: From Pip to uv

Switching an existing project from Pip to uv is quite straightforward. Below is a quick reference table showing how common pip commands map to their uv equivalents:

Pip Command	uv Equivalent	Description
`python -m venv .venv`	`uv venv`	Creates a virtual environment
`pip install package`	`uv add package`	Installs a package and updates project files
`pip install -r requirements.txt`	`uv pip install -r requirements.txt`	Installs from requirements file
`pip freeze > requirements.txt`	`uv export -o requirements.txt`	Exports dependencies
`pip list`	`uv pip list`	Lists installed packages
`pip uninstall package`	`uv remove package`	Removes a package

If your project currently uses pip and requirements.txt, you can migrate it to uv in just a few steps.

Initialize uv in the existing project directory:
```
uv init .
```
This creates a pyproject.toml file and sets up the project structure, without replacing any existing files.
Remove the old virtual environment:
```
rm -rf .venv
```
Install dependencies from requirements.txt:
```
uv add -r requirements.txt
```
This installs the dependencies listed in your requirements.txt file in a virtual environment, and updates the pyproject.yml and uv.lock files accordingly.
Create a new uv-managed environment:
```
uv sync
```
Verify that everything works — for example (assuming you have a Django project):
```
uv run manage.py migrate
```
```
uv run manage.py runserver
```

Now your project should be fully migrated to uv with the same functionality as before, but now with uv's performance advantages.

Wrapping Up

By unifying the roles of multiple tools into one high-performance system, uv eliminates much of the friction developers have long dealt with.

Its speed alone is a compelling reason to adopt it, but uv's streamlined workflow and modern packaging standards make it even more appealing. For most projects, the transition is simple and the productivity gains are immediate.

uv's development is a strong signal of where the broader Python ecosystem is headed. Now is an ideal time to adopt this tool built for the future.

Thanks for reading!

Scheduling Background Tasks in Python with Celery and RabbitMQ

Roy Tomeij — 2025-08-27T00:00:00+00:00

It's important and useful to schedule background tasks for your Python application. Tasks allow your app to perform time-based or long-running operations without blocking the main thread or slowing down the user-facing functionality of your app.

Background tasks can be used for anything from running recurring jobs like data cleanup or reporting, to sending asynchronous emails or other notifications.

In this article, we will build background tasks using Celery and RabbitMQ to create a weather notification service that will deliver rain alerts with Slack.

But first, let's briefly explore why we would want to use Celery or RabbitMQ in the first place.

Why Celery?

Celery is a widely used distributed task queue framework for Python applications. Celery handles distributing background tasks to worker processes or nodes, manages their execution reliably, and can scale to hundreds or thousands of jobs per second in production environments.

Celery is especially popular in the Python ecosystem because it fits naturally with the language’s syntax, and it's commonly used in frameworks like Django or Flask. Celery uses familiar concepts — decorators for tasks, Python data structures for messages — and integrates easily with Python’s logging, error handling, and testing tools. Its configuration is also Python-based, avoiding the need to learn an entirely separate domain-specific language.

Lastly, Celery’s architecture is highly flexible. It works with different message brokers such as RabbitMQ (which we will use in this tutorial), Redis, or Amazon SQS, letting the developer pick a broker that best suits their workload and infrastructure.

Why RabbitMQ?

RabbitMQ is a robust, battle-tested message broker known for its reliability and flexibility. It supports complex routing patterns via exchanges and queues, offers strong delivery guarantees (including persistence and acknowledgments), and has excellent interoperability across languages and protocols (e.g., AMQP, MQTT, STOMP).

Despite its strengths, like Celery, RabbitMQ can be complex to set up and maintain compared to alternatives like Redis or cloud-native solutions like AWS SQS. Also, RabbitMQ stores messages on disk by default, which can introduce latency. For simple pub/sub or fire-and-forget patterns, lighter brokers like NATS or Redis Streams might perform better and could be easier to manage.

Now let's move on to building our weather notification service.

What We Will Build

In this practical example, we will create a Slack rain notification service that will check the weather forecast every day at 12pm to see whether rain is expected for the next 5 days.

We'll need to add 3 different APIs to our project: the OpenWeatherMap API (for getting our rain data), the GitHub API (which will act as a simple JSON database to ensure we don't get duplicate notifications), and the Slack API for creating messages.

In the last section of this article, we will integrate Celery and RabbitMQ into our application to create the automatic 12pm daily task schedule. Find the full code for this tutorial on GitHub.

The file structure of our Python project will look like the diagram below. main.py is the main entrypoint for our app and will include the code for making the initial request to the OpenWeatherMap API. In the apis folder, we have two separate files — github_data.py and slack.py. These handle saving data to GitHub and creating messages in Slack.

.
├── apis
│   ├── github_data.py
│   └── slack.py
├── celerybeat-schedule
├── main.py
└── requirements.txt

Prerequisites and Requirements

The steps in this project have been created on macOS running Apple's M2 chip. In particular, later in the Implement Celery and RabbitMQ section, I install and verify RabbitMQ using Homebrew for Mac. But since RabbitMQ is cross-platform, you can also use Linux and Windows. See the official RabbitMQ documentation for more details on installation using other operating systems.
This project uses Python 3.10.7, but you can also safely use any later Python version. Everything will be installed with pip inside a virtual environment.
You need basic knowledge of Python, experience working with APIs in Python, and some basic knowledge of terminal commands.
You need knowledge of GitHub token creation and setting repository permissions.

Initial Python Project Setup

First, let's create a Python project in a new directory called python-celery-rabbitmq. We will also create the core main.py file.

mkdir python-celery-rabbitmq
cd python-celery-rabbitmq
touch main.py

Next, before we start installing our packages with pip, we need to set up our virtual environment. First run:

python -m venv .venv

Activate your virtual environment:

source .venv/bin/activate

Now we can install all our required packages:

pip install requests celery python-dotenv

In the next section, we will add the code in main.py to make requests to the OpenWeatherMap API.

Query the OpenWeatherMap API with a New Python Project

Head to the OpenWeatherMap API page, create a new account, and subscribe to the cheapest pay-as-you-go plan called: "One Call API 3.0". The website has a number of other monthly subscription and professional-tier products below the basic "One Call API 3.0" plan. But we can safely ignore those options for the purposes of this article.

Also, confusingly, the OpenWeatherMap website says that the first 1000 API calls are free for One Call API. But once you go to the Stripe payment page, it says only the first 100 calls are free. Anyway, whatever the case, even with a lot of project testing, we should stay way below this limit.

Once we are subscribed, grab the API key from your dashboard, and we can start the actual coding!

At this point, it's important that we don't expose our API key. Let's add the key to our .env file and also make sure the .env file has been added to the project's .gitignore so it is not tracked in version control.

In main.py, add the code below (note: this will change once we implement the Celery functionality in a later section of this post):

import os
import requests
from celery import Celery
from celery.schedules import crontab
from apis.github_data import check_github_json
from dotenv import load_dotenv
load_dotenv()

def weather_task(city):
    api_key = str(os.getenv('OPEN_WEATHER_MAP_API_KEY'))
    url = "https://api.openweathermap.org/data/2.5/forecast"
    params = {"q": city, "appid": api_key, "units": "metric"}
    response = requests.get(url, params=params)

    if response.status_code != 200:
        print(f"Error {response.status_code}: {response.text}")
        return

    data = response.json()
    summaries = []

    for entry in data["list"]:
        if "12:00:00" in entry["dt_txt"]:
            main = entry["weather"][0]["main"]
            description = entry["weather"][0]["description"]

            if "rain" in description.lower() or "rain" in main.lower() or "rain" in entry:
                date = entry["dt_txt"].split(" ")[0]
                summaries.append({
                    "Date": date,
                    "Weather Description": description
                })

    if summaries:
        check_github_json(summaries)
    else:
        check_github_json(None)

if __name__ == "__main__":
    weather_task("London")

There are three main things to break down in this code snippet:

First of all, we should note that the request is using the API's 5-day forecast endpoint. Since this endpoint returns multiple forecasts in three-hour increments over the 5 days, we need to reduce the amount of data returned. We can do this by just getting a "summary" for the forecast at 12pm, which simplifies things greatly and is sufficient for our purposes here. So we use an if statement to only filter entries containing 12:00:00.
Next, we need to find any instances of rain in the API data. So we'll use another three-part if statement to look for any mention of rain in three possible parts of the data: if "rain" in description.lower() or "rain" in main.lower() or "rain" in entry. If rain is indeed found, we append the date and a plain language weather description to the summaries list.
Lastly, we have one final, important if statement. If we have some rain summary data, we will pass this through to the check_github_json function. But if not, we will just pass None into that function.

Let's see exactly what check_github_json does and how it works together with the other two associated functions in the github_data.py file.

Use the GitHub API to Automatically De-duplicate Notifications

As mentioned earlier, we will be using the GitHub API as a convenient/no-frills database for storing JSON data from the OpenWeatherMap API every time there is a new or unique rain forecast. By storing the data with GitHub, we can check any incoming forecasts from the OpenWeatherMap API and ensure that the new forecasts are actually unique. This will allow us to prevent any duplicate notifications from taking place on Slack.

Create an empty GitHub repository called Weather Data. Add a folder called json and then create a file called data.json inside that. Inside data.json, add an empty array (identical to a Python list): [].

Next, create your GitHub public access token for your repository and set the repository permissions.

Copy the token and add it to your .env file with the key GITHUB_PERSONAL_ACCESS_TOKEN:

GITHUB_PERSONAL_ACCESS_TOKEN=your-long-alpha-numeric-github-token-here

Back in our code editor, let's add all the import statements and three functions, called check_github_json, load_github_json, and update_github_json, in a new file called github_data.py:

import os
import logging
import json
from github import Github
from datetime import datetime, timezone
from dotenv import load_dotenv
load_dotenv()
logger = logging.getLogger(__name__)

def load_github_json():
    GITHUB_PERSONAL_ACCESS_TOKEN = str(os.getenv('GITHUB_PERSONAL_ACCESS_TOKEN'))
    github = Github(GITHUB_PERSONAL_ACCESS_TOKEN)
    try:
        repo = github.get_user().get_repo('weather-data')
        file = repo.get_contents("json/data.json")
        db_data = json.loads(file.decoded_content.decode('utf-8'))
        return repo, file, db_data
    except Exception as e:
        logger.exception("An error occurred")


def check_github_json(forecast):
    if forecast is None:
        return

    repo, file, db_data = load_github_json()
    new_items = []
    is_first_run = len(db_data) == 0

    for item in forecast:
        if item not in db_data:
            print("Make JSON DB entry")
            db_data.append(item)
            new_items.append(item)
        else:
            print("Skip, found in DB")

    if new_items:
        if is_first_run:
            update_github_json(repo, file, db_data, slack_items=new_items)
        else:
            update_github_json(repo, file, db_data, slack_items=[new_items[-1]])
    else:
        print("No new items to commit")

def update_github_json(repo, file, updated_data, slack_items):
    bytes_data = json.dumps(updated_data).encode('utf-8')
    commit_msg = datetime.now(timezone.utc).strftime("Update weather data - %Y-%m-%d %H:%M:%S UTC")
    try:
        repo.update_file(file.path, commit_msg, bytes_data, file.sha)
        print("Github data updated!")
        # create_slack_message(slack_items)
    except Exception as e:
        logger.exception("An error occurred")

First, take a quick look at the load_github_json function. The code inside this function makes the request to get our weather_data repository, then we get the content of our data.json file and load the file as JSON. The variables repo, file, and db_data can then be reused in check_github_data and update_github_data functions.

In the check_github_json function, we use a for loop to iterate through the forecast data we received from the OpenWeatherMap data in main.py. First, we check if the forecast item is already in our GitHub db_data JSON. If it's not, we append to db_data and the new_items variables. For the first run, we want to send all our new_items to the update_github_json function. On subsequent runs, we just want to pass the last item (using -1) to that function.

Lastly, in update_github_json, we execute the built-in update_file method, which requires a commit message like all changes to a GitHub repository. Here, we create a unique commit message each time by recording the exact time and date of the commit using Python's datetime module.

At this point, let's check that everything is working as intended by running python main.py in our project root.

If we are successful, first, we should see something similar to this printed in our terminal:

Make JSON DB entry
Make JSON DB entry
Make JSON DB entry
Github data updated!

This is already a pretty good sign that things have worked correctly! Now, go to your GitHub repository and click on the Raw button to see your data more clearly. Your empty array should be replaced with the new weather data:

[
  {
    "Date": "2025-06-05",
    "Weather Description": "light rain"
  },
  {
    "Date": "2025-06-07",
    "Weather Description": "light rain"
  },
  {
    "Date": "2025-06-08",
    "Weather Description": "light rain"
  }
]

In our next section on the Slack API, we will implement code to fire off a notification once we get a new and unique rain forecast.

Create Message Notifications with the Slack API

In this section, we will use the Slack WebClient SDK.

First, let's go to the Slack API page to get our "Bot" credentials so we can implement our notifications feature:

Go to https://api.slack.com/apps, click on Create New App and select From Scratch.
In the next modal window, give your app a name and select the workspace you want to deploy it to (it's a good idea to create your own private workspace dedicated to this project).
Click on OAuth & Permissions, and then under Bot Token Scopes, update the following three permissions: channels:join, channels:read, and chat:write.

(It will soon become clear why we need the channel permissions in addition to the more obvious chat:write permission).

Lastly, near the top of the page, click on Install to Workspace. This will generate your new Bot User OAuth Token. Again, copy the token and add it to your .env file with the key SLACK_BOT_USER_TOKEN:

SLACK_BOT_USER_TOKEN=your-long-alpha-numeric-bot-token-here

Now we can create a new file in the project root called slack.py and add the code below:

import os
from slack_sdk import WebClient
from dotenv import load_dotenv
load_dotenv()

def create_slack_message(items):
    SLACK_BOT_TOKEN = str(os.getenv('SLACK_BOT_USER_TOKEN'))
    client = WebClient(token=SLACK_BOT_TOKEN)
    channel = "C0211DW58JD"

    if isinstance(items, str):
        text_to_send = items
    else:
        message_lines = [f"- {item['Date']}: {item['Weather Description']}" for item in items]
        text_to_send = "New rain alert(s):\n" + "\n".join(message_lines)

    try:
        client.conversations_join(channel=channel)
    except Exception:
        pass

    client.chat_postMessage(channel=channel, text=text_to_send)

After instantiating the WebClient with our SLACK_BOT_USER_TOKEN, we specify the Slack channel we are going to use for our notifications. This must be in ID form as the Slack SDK doesn't understand plain language channel names. So first, in order to get your own unique channel ID, you need to temporarily run the for loop below in your code (if you use the channel ID above, it will not work for your own unique installation):

# response = client.conversations_list()
# for channel in response['channels']:
#     print(channel['name'], channel['id'])

Next, the if isinstance(items, str) line checks if the items variable is a plain language string to simply pass through the strings: "No rain expected in the next 5 days" or "No change in the forecast found over the next 5 days". If it's not a string, the code will process the list of dictionaries and display it as a nicely-formatted message for the Slack notification.

The next try/except block is just a convenience measure to ensure that the bot is a member of the channel. If it's not, it makes sure that the bot can automatically join the channel. This saves us from having to manually add the bot as a channel member in the Slack UI (if the bot is not a channel member, we will get errors).

Lastly, with just the line client.chat_postMessage(channel=channel, text=text_to_send), we will execute the chat_postMessage and actually create the message notification.

Now let's go back to github_data.py, import the create_slack_message function, and call it in the three places inside check_github_json and update_github_json, as shown below (annotated with #New for clarity):

import os
import logging
import json
from github import Github
#New
from apis.slack import create_slack_message
from datetime import datetime, timezone
from dotenv import load_dotenv
load_dotenv()
logger = logging.getLogger(__name__)

...

def check_github_json(forecast):

    if forecast is None:
        #New
        create_slack_message("No rain expected in the next 5 days.")
        return

    repo, file, db_data = load_github_json()
    new_items = []
    is_first_run = len(db_data) == 0

    for item in forecast:
        if item not in db_data:
            print("Make JSON DB entry")
            db_data.append(item)
            new_items.append(item)
        else:
            print("Skip, found in DB")

    if new_items:
        if is_first_run:
            update_github_json(repo, file, db_data, slack_items=new_items)
        else:
            update_github_json(repo, file, db_data, slack_items=[new_items[-1]])
    else:
        print("No new items to commit")
        #New
        create_slack_message("No change in the forecast found over the next 5 days.")


def update_github_json(repo, file, updated_data, slack_items):
    bytes_data = json.dumps(updated_data).encode('utf-8')
    commit_msg = datetime.now(timezone.utc).strftime("Update weather data - %Y-%m-%d %H:%M:%S UTC")
    try:
        repo.update_file(file.path, commit_msg, bytes_data, file.sha)
        print("Github data updated!")
        #New
        create_slack_message(slack_items)
    except Exception as e:
        logger.exception("An error occurred")

Also, it might be a good idea to revert back to an empty array in our GitHub weather_data repository, so we can run our next test from a clean starting point.

Once that's done, run python main.py again.

If everything is working smoothly, this should trigger a new Slack message notification:

Implement Celery for Python and RabbitMQ

Before we can start work on our Celery background task system, we need to make sure RabbitMQ is installed and running on our system.

For Mac, the easiest way to install RabbitMQ is with Homebrew. To install on other operating systems, see the official RabbitMQ documentation.

First, run these two installation commands:

brew install erlang
brew install rabbitmq

To start RabbitMQ as a background service, run this command:

brew services start rabbitmq

You can also stop RabbitMQ at any time with:

brew services stop rabbitmq

Since we already installed Celery in our project with pip earlier, we can now move onto editing main.py and adding the Celery functionality. Here is our new main.py, with comments indicating where new code has been added:

import os
import requests
from celery import Celery
from celery.schedules import crontab
from github_data import check_github_json
from dotenv import load_dotenv
load_dotenv()

# New Celery Code:
app = Celery('main', broker='amqp://localhost')
app.conf.timezone = 'Europe/London'
app.conf.broker_pool_limit = 1
app.conf.beat_schedule = {
    'check-weather-daily-at-noon': {
        'task': 'main.weather_task',
        'schedule': crontab(hour=12, minute=0),
        'args': ('London',)
    },
}

# New decorator
@app.task(name='main.weather_task')
def weather_task(city):
    api_key = str(os.getenv('OPEN_WEATHER_MAP_API_KEY'))
    url = "https://api.openweathermap.org/data/2.5/forecast"
    params = {"q": city, "appid": api_key, "units": "metric"}
    response = requests.get(url, params=params)

    if response.status_code != 200:
        print(f"Error {response.status_code}: {response.text}")
        return

    data = response.json()
    summaries = []

    for entry in data["list"]:
        if "12:00:00" in entry["dt_txt"]:
            main = entry["weather"][0]["main"]
            description = entry["weather"][0]["description"]

            if "rain" in description.lower() or "rain" in main.lower() or "rain" in entry:
                date = entry["dt_txt"].split(" ")[0]
                summaries.append({
                    "Date": date,
                    "Weather Description": description
                })

    if summaries:
        check_github_json(summaries)
    else:
        check_github_json(None)

# Below used before Celery, commented-out now but kept if you want to test the API without Celery
# if __name__ == "__main__":
# 	weather_task("London")

The main code that's of interest in the above snippet is the block labelled "# New Celery Code". On the first line of this block, we create a new Celery application instance and connect it to the local RabbitMQ message broker using the AMQP protocol. Under this line, I've set the timezone to 'Europe/London' (since this is where I'm based), but you should change this to your own timezone.

Next, we limit the number of RabbitMQ broker connections to 1 with app.conf.broker_pool_limit = 1. (Note: you can increase this connection number if you encounter performance issues down the line.)

With app.conf.beat_schedule, we create a configuration dictionary for our Celery scheduler. We call our overall "job": check-weather-daily-at-noon, specify that the task we want to run is the main.weather_task function, and, using crontab, set the task to run every day at 12pm. Finally, the args key passes "London" as an argument to the function, so we receive updates on London weather.

Now, instead of running python main.py, we run our Celery beat and worker commands in one line:

celery -A main worker --loglevel=info --beat

If everything is working correctly, you should see something similar to the following output:

-------------- celery@Daniels-MacBook-Pro-4.local v5.5.2 (immunity)
--- ***** -----
-- ******* ---- macOS-10.16-x86_64-i386-64bit 2025-06-06 12:14:23
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         main:0x10c163a60
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . main.weather_task

In order to be 100% sure that the task will actually fire, you can play around with the hour and minute settings to make it close to your current time (which should finally trigger the message notification in Slack).

And that's it!

Wrapping Up

If you've made it this far, well done — we've done a lot of work together!

In this tutorial, we brought together Celery, RabbitMQ, and three external APIs to create an automated rain notification service. We learned how to fetch weather forecasts with OpenWeatherMap, avoid duplicate alerts using GitHub as a lightweight JSON store, and deliver notifications via the Slack API. Finally, we used Celery and RabbitMQ to run the task automatically every day at noon.

Whether you're monitoring weather, syncing data, or triggering reminders, this project can serve as a foundation to help you build more background automation tasks in the near future.

Happy automating!

How to Use Redis with Python

Roy Tomeij — 2025-08-20T00:00:00+00:00

When it comes to data-driven applications, developers and data engineers are always trying to balance factors such as scalability, speed, flexibility, latency, and availability.

In other words, databases and infrastructure are the foundations for well-structured applications: just like bricks are for houses.

This article explores Redis' data store features and includes use cases. We'll learn how to use Redis in Python with a step-by-step tutorial.

Let's get started!

What Is Redis?

Redis (short for Remote Dictionary Server) is an open-source, in-memory data structure store that can be used as a database, cache, message broker, or queue. It is known for its high performance, low latency, and versatility as it stores data in memory (on the RAM) rather than on disk, which makes it extremely fast for read and write operations.

Redis is often referred to as a data structure server because it provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. This means that different processes can query and modify the same data structures in a shared way.

Why Use Redis?

Before describing Redis' most common use cases, let's explore its main features:

In-memory storage: In Redis, data is stored in RAM, providing response times in sub-milliseconds.
Persistence options: While data is stored in memory, Redis also offers persistance options like RDB (Redis Database Backup) — periodic dataset snapshots — and AOF (Append-Only File), which logs every write operation for durability. This allows data to be saved to disk and replicated across multiple servers to ensure data durability and availability.
Pub/sub messaging: It supports publish/subscribe messaging patterns for real-time communication.
Clustering and replication: Redis supports clustering and primary-secondary replication, providing scalability and high availability.

Finally, at its core, Redis is a key-value store, meaning it stores data as a collection of keys and their associated values.

A key in Redis is a unique string that acts as an identifier for the associated value. Keys are case-sensitive and can contain any binary sequence (e.g., strings, numbers, or special characters).

A value in Redis can be one of several data types:

Strings
Hashes
Lists
Sets
Sorted Sets
Bitmaps
HyperLogLogs
Streams

Redis provides a wide set of commands for working with key-value pairs, such as SET, GET, and DEL for strings, HSET, HGET, and HDEL for hashes, and LPUSH, LGET, and LREM for lists. Here are some examples of key-value pairs:

SET user:1001 "John Doe"
HSET user:1001 name "John Doe" age 30 email "john@example.com"
LPUSH tasks "task1" "task2" "task3"

The management of such data structures in Redis is important because:

Redis stores them on disk, even if they are always served and modified into server memory. This means that Redis is fast, but also non-volatile.
The implementation of data structures emphasizes memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modelled using a high-level programming language.
Redis offers a number of features commonly found in databases, such as replication, tunable durability levels, clustering, and high availability.

With this in mind, here are some common applications of Redis:

Caching: As it stores data in memory, one of the main uses of Redis is as a caching system for web applications, since it reduces server load and improves page loading times.

Real-time applications: Given its speed and reliability, Redis is perfect for real-time use cases like chat applications, real-time analytics, and live notifications.
Session management: Another common use of Redis is storing user session data in web applications because of its speed and ability to handle expiration policies.
Message queues: Redis can act as a lightweight message broker using its pub/sub or streams features, making it suitable for event-driven architectures.
Advanced data structures management: Redis supports advanced data structures that are not available in traditional databases, making it useful for tasks like counting unique visitors (HyperLogLog), storing time-series data (Streams), and implementing priority queues (Sorted Sets).

How to Use Redis in Python: A Step-By-Step Tutorial

Now that we've covered the theory, it's time to see how to use Redis in Python.

Prerequisites

To replicate this tutorial, your system must have the following prerequisites:

Python 3.6 or higher: Any Python version higher than 3.6 will do. Specifically, we will install dependencies via pip (already included in any Python version from 3.4+).

Step 1: Setting Up the Environment and Installing Dependencies

Suppose you call the main folder of your project redis. The project structure will be as follows:

redis/
│
├── venv/
└── examples/
   ├── basic_connection.py
   ├── strings_example.py
   ├── hashes_example.py
   ├── sets_example.py
   ├── transactions_example.py
   └── expiration.py

Where:

The .py files will contain the logic described in the next steps.
venv/ contains the virtual environment.

You can create the venv/ virtual environment directory like so:

python -m venv

To activate it on Windows, run:

venv\Scripts\activate

On macOS/Linux, execute:

source venv/bin/activate

In the activated virtual environment, install Redis:

pip install redis

Step 2: Testing The Connection to Redis

To establish a connection to a Redis server using redis-py, type the following code into the basic_connection.py file:

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

print(r.ping())

When you run it via python3 basic_connection.py, if everything works fine, you will receive:

True

Step 3: Managing Data Structures

We've learned that Redis stores data as key-value pairs, and the values can be different data structures. Now it's time to see how this works.

To manage strings, type the following in strings_example.py:

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

r.set('user:1001', 'John Doe')
print(r.get('user:1001'))

Which, as expected, returns:

John Doe

To manage hashes, type the following in the hashes_example.py file:

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

r.hset('user:1', 'name', 'Alice')
print(r.hget('user:1', 'name'))

Which returns:

Alice

To manage sets, write the following into the sets_example.py file:

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

r.sadd('tags', 'python', 'redis')
print(r.smembers('tags'))

Which returns:

{'redis', 'python'}

Step 4: Using Advanced Redis Features

Redis provides various advanced features, including pipelines.

A pipeline in Redis is a way to batch multiple commands together and send them to the Redis server in a single network request. This reduces the overhead of multiple round trips between the client and the server, improving performance, especially when executing multiple commands.

For example, if you want to create a pipeline object (pipe) that queues commands instead of executing them immediately, type the following into the transactions_example.py file:

import redis
r = redis.Redis(host='localhost', port=6379, db=0)

with r.pipeline() as pipe:
    pipe.set('key1', 'value1')
    pipe.set('key2', 'value2')
    pipe.execute()

The pipe.execute() method sends all the queued commands in the pipeline to the Redis server in a single request. Then, the server processes the commands in the order they are queued.

Another advanced feature of Redis is the possibility to set keys with expiration times, retrieve them before and after expiration, and handle scenarios where the keys no longer exist. To do so, write the following into the expiration.py file:

import time
import redis

# Connect to the Redis server
r = redis.Redis(host='localhost', port=6379, db=0)

# Set keys with expiration using setex()
print("Setting keys with expiration...")
r.setex('temp_key1', 5, 'value1')  # Expires in 5 seconds
r.setex('temp_key2', 10, 'value2')  # Expires in 10 seconds

# Retrieve the keys before expiration
print("\nRetrieving keys before expiration:")
print(f"temp_key1: {r.get('temp_key1')}")
print(f"temp_key2: {r.get('temp_key2')}")

# Wait for 6 seconds
print("\nWaiting for 6 seconds...")
time.sleep(6)

# Try retrieving the keys after expiration
print("\nRetrieving keys after 6 seconds:")
print(f"temp_key1: {r.get('temp_key1')}")
print(f"temp_key2: {r.get('temp_key2')}")

# Wait for another 5 seconds
print("\nWaiting for another 5 seconds...")
time.sleep(5)

# Try retrieving the keys again
print("\nRetrieving keys after 11 seconds:")
print(f"temp_key1: {r.get('temp_key1')}")
print(f"temp_key2: {r.get('temp_key2')}")

# Set a key with expiration and check its TTL
print("\nSetting a new key with expiration...")
r.setex('temp_key3', 15, 'value3')
print(f"temp_key3: {r.get('temp_key3')}")
print(f"TTL for temp_key3: {r.ttl('temp_key3')} seconds")

# Wait for 5 seconds and check TTL again
print("\nWaiting for 5 seconds...")
time.sleep(5)
print(f"TTL for temp_key3 after 5 seconds: {r.ttl('temp_key3')} seconds")

# Delete the key before it expires
print("\nDeleting temp_key3 before it expires...")
r.delete('temp_key3')
print(f"temp_key3: {r.get('temp_key3')}")

Here is what this code does:

The setex(name, time, value) method sets a key with a specific expiration time (in seconds). In the provided example, the keys have different expiration times.
The get() method retrieves the value of a key. Before the expiration time, the keys return their respective values.
The ttl() method retrieves the remaining time (in seconds) before a key expires.

Here is the expected result after the whole process is completed:

Setting keys with expiration...

Retrieving keys before expiration:
temp_key1: b'value1'
temp_key2: b'value2'

Waiting for 6 seconds...

Retrieving keys after 6 seconds:
temp_key1: None
temp_key2: b'value2'

Waiting for another 5 seconds...

Retrieving keys after 11 seconds:
temp_key1: None
temp_key2: None

Setting a new key with expiration...
temp_key3: b'value3'
TTL for temp_key3: 15 seconds

Waiting for 5 seconds...
TTL for temp_key3 after 5 seconds: 10 seconds

Deleting temp_key3 before it expires...
temp_key3: None

How to Use Redis in Python for Monitoring

Monitoring a Redis server is an essential aspect of managing a production system, for more than just checking general statistics.

Redis offers built-in methods for monitoring, like:

info(): Providing comprehensive details about memory usage, client connections, CPU utilization, and other important statistics.
monitor(): Streaming every command processed by the Redis server in real time. (However, consider that using this in a production environment can be resource-intensive and may impact performance.)
client_list(): Returning details about connected clients, which is useful for identifying unusual client behavior or ensuring your connection pools are behaving as expected.
slowlog_get(): Allowing you to track slow-executing commands to pinpoint potential bottlenecks or inefficient queries.

Here's how to implement the above monitoring methods:

import redis
import threading
import time

def server_stats_monitor():
    """
    Periodically retrieves and prints key Redis server statistics.
    It uses INFO to extract memory usage, client counts, CPU usage, and uptime.
    """
    r = redis.Redis(host='localhost', port=6379, db=0)

    while True:
        info = r.info()
        print("=== Redis Server Monitoring ===")
        print("Used Memory:", info.get("used_memory_human"))
        print("Connected Clients:", info.get("connected_clients"))
        print("CPU Usage:", info.get("used_cpu_sys"), "system CPU seconds,",
              info.get("used_cpu_user"), "user CPU seconds")
        print("Uptime (seconds):", info.get("uptime_in_seconds"))
        print()  # Blank line for readability

        # Retrieve and display slow log entries (if any)
        slow_logs = r.slowlog_get()
        print("=== Slow Log Entries ===")
        if slow_logs:
            for log in slow_logs:
                # Each log entry is a dictionary with keys like 'id', 'start_time', 'duration', and 'command'
                print(f"ID: {log.get('id')}, Duration: {log.get('duration')} μs, Command: {log.get('command')}")
        else:
            print("No slow log entries found.")
        print()

        # Retrieve and display client list details
        client_list = r.client_list()
        print("=== Connected Clients ===")
        for client in client_list:
            # Displaying each client info (address, age, idle time, flags, db, etc.)
            print(f"Address: {client.get('addr')} | Idle Time: {client.get('idle')} sec | DB: {client.get('db')}")
        print("\n-----------------------------\n")

        time.sleep(10)  # Pause for 10 seconds before repeating

# Start the monitoring in a background thread so that it runs continuously.
stats_thread = threading.Thread(target=server_stats_monitor)
stats_thread.daemon = True
stats_thread.start()


def monitor_commands():
    """
    Sets up a real-time monitor that listens to every command processed by Redis.
    Note: Running MONITOR can be heavy on performance and should be used cautiously.
    """
    r = redis.Redis(host='localhost', port=6379, db=0)
    monitor = r.monitor()
    for command in monitor.listen():
        print("Real-time Command:", command)


# Keep the main thread active so the background monitoring continues.
while True:
    time.sleep(1)

This code does the following:

It prints server statistics every ten seconds using the info() method.
slowlog_get() gets slow log entries, if any. Each entry is typically a dictionary containing an identifier, start time, duration (in microseconds), and the command that ran slowly. This helps you to identify any commands that might be causing performance issues.
The client_list() gets the connected clients' status. The returned result is a list of dictionaries, each containing details such as the client's address, idle time, connected database, and other flags. Monitoring these details can help you find issues with idle or misbehaving clients.
An optional real-time monitor using monitor() is provided to get real time data.

The expected result is something like this:

=== Redis Server Monitoring ===
Used Memory: 852.59Kz\
Connected Clients: 1
CPU Usage: 2.931738 system CPU seconds, 1.491924 user CPU seconds
Uptime (seconds): 1773

=== Slow Log Entries ===
No slow log entries found.

=== Connected Clients ===
Address: xxx.x.x.x:xxxxx | Idle Time: 0 sec | DB: 0

But, of course, it will continue to provide statistics in a loop:

Monitoring Redis with AppSignal

While Redis provides built-in monitoring features, they are low-level, infrastructure-focused metrics for understanding raw performance, resource consumption, and debugging specific issues within the Redis server itself.

If you want to develop a more detailed understanding of your application's overall health, you can use AppSignal's integration with Redis.

AppSignal provides performance metrics, error tracking, and logging for your applications.

To use AppSignal, first sign up for a 30-day free trial, use the Python installation guide and configuration guide.

Then you can integrate AppSignal with Redis like so:

import redis
import threading
import time
from appsignal import Appsignal, probes, set_gauge

# Initialize the AppSignal client with your AppSignal push API key and application name.
appsignal = Appsignal(
    push_api_key="YOUR_APPSIGNAL_PUSH_API_KEY",
    name="redis_monitor",
    active=True
)

def monitor_redis_with_appsignal():
    """
    This function collects Redis monitoring data and sends selected metrics
    to AppSignal using its Python API. Metrics include memory usage, client counts,
    uptime, slow log details, and more.
    """
    # Connect to your Redis server
    r = redis.Redis(host='localhost', port=6379, db=0)

    # Collect basic server stats using the INFO command
    info = r.info()
    metrics = {
        "redis.used_memory_bytes": info.get("used_memory"),
        "redis.used_memory_human": info.get("used_memory_human"),
        "redis.connected_clients": info.get("connected_clients"),
        "redis.cpu_sys": info.get("used_cpu_sys"),
        "redis.cpu_user": info.get("used_cpu_user"),
        "redis.uptime_seconds": info.get("uptime_in_seconds")
    }

    # Use the AppSignal API to record each metric.
    for metric_name, value in metrics.items():
        set_gauge(metric_name, value)

probes.register("redis", monitor_redis_with_appsignal)

This example instantiates the Appsignal class to send Redis monitoring metrics directly to AppSignal. This approach allows you to:

Correlate Redis data with transaction traces and error reports from the rest of your application. For example, if a spike in redis.slowlog_count coincides with a surge in request latency or error notifications, you have immediate insight into a potential bottleneck or failure point.
Visualize metric history over time, as AppSignal stores historical data. This allows you to identify trends, seasonal variations, or gradual degradations in Redis performance — insights that are not as obvious when only using Redis’s native monitoring features.
Set up custom alerts in AppSignal that are triggered when certain thresholds are breached.

And that's it for this post!

Wrapping Up

In this article, you learned what Redis is, why you should use it, and how to use it with Python.

We also explored how to use Redis' monitoring features and set up AppSignal to monitor Redis for more in-depth insights into your application.

Happy coding!

Deploy a Python Flask App to Render with Docker

Roy Tomeij — 2025-08-06T00:00:00+00:00

In this tutorial, we will build a Flask app with Docker, create a background worker with Celery, and use RabbitMQ as the message broker between the two services. This (relatively) simple example will be used to demonstrate how Docker makes it easier to run multiple services and share configuration details between developers.

In the last section, we will also build and optimize our app for deployment to Render, so we can avoid some of the common gotchas it is easy to fall victim to along the way!

First, let's quickly touch on why we're using Flask, Docker, and Render.

Why Flask?

Flask is a lightweight and flexible Python web framework that's ideal for small to medium-sized applications. Flask provides both a high degree of simplicity and extensibility via Python's rich package library. This makes it a great choice for rapid prototyping and microservices.

On the downside, Flask lacks built-in features found in more comprehensive frameworks like Django. This means you will need to integrate common app functionality like user authorization/authentication, form handling, and database management from scratch.

Why Docker?

Docker allows developers to wrap applications with dependencies into a single virtual container, ensuring consistency across development, testing, and production environments. This makes it easier to manage complex setups, simplifies deployment, and avoids the typical developer refrain: "but it works on my machine". Docker also integrates well with CI/CD pipelines and supports orchestration tools like Docker Compose and Kubernetes.

But Docker does introduce an additional layer of complexity and a steeper learning curve. Resource usage can also be higher compared to running an app directly on your local machine without a virtual container.

Why Render?

Render offers a developer-friendly platform for deploying web apps, APIs, static sites, background workers, and more. It automates deployment from Git repositories, provides free SSL, custom domains, and offers managed services like PostgreSQL and Redis. Render’s simplicity and pay-as-you-go pricing model are particularly suited for startups and small developer teams.

That said, Render has limitations in the level of customization offered compared to more mature platforms like AWS or GCP. For teams with specific infrastructure needs or strict performance requirements, you might need to consider something else.

Let's turn to installing Docker on our machine next.

Docker Installation

In this section, we will not follow the recommended approach of installing Docker Desktop for macOS. At the time of writing, my laptop running macOS Ventura with Apple's M-series silicon chip flags a false security certificate issue. Windows users can also check out the official documentation, but please note this hasn't been battle-tested like the following macOS installation steps.

The Docker Desktop security issue is not just alarming in itself, but also a major headache! Of course, there is a fix: manually adding the certificate via the command line and removing the (rather dramatic) macOS warning. But even after doing this, the Mac operating system appears to silently block the Docker Desktop app from launching. This is very frustrating, to say the least!

So, to save you from all the drama, I recommend installing the necessary Docker app components manually from the command line (Docker Desktop automatically bundles all these individual components together). Let's go through the steps one by one.

First, use Homebrew to install Docker's main command line interface (CLI) application:

brew install docker

Next, install Colima, a lightweight Docker virtual machine (VM) which works well on Macs with M1 and M2 chips:

brew install colima

Then run:

colima start

As a result, you should see this output:

INFO[0001] starting colima
INFO[0001] runtime: docker
INFO[0002] starting ...                                  context=vm
INFO[0015] provisioning ...                              context=docker
INFO[0016] starting ...                                  context=docker
INFO[0018] done

You can stop Colima at any point with:

colima stop

To run the docker compose up command in the next section when we build our sample Flask app, we need to install docker-compose:

brew install docker-compose

Lastly, if you have previously attempted to install Docker Desktop (like me!), you will need to separately install the Docker credentials helper component. You will also have to remove any old references to how Docker Desktop handles credentials in one of the configuration files.

So now run this command:

brew install docker-credential-helper

Open the hidden configuration file: ~/.docker/config.json and make sure the following line in the JSON file looks like this:

"credsStore": "osxkeychain"

You can test that everything is working correctly by running Docker's built-in "hello world" command:

docker run hello-world

Hopefully, you will get the output below, and we can start building!

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Create Your Python Flask App With Docker

First, let's create a new project directory called flask-docker-render and an app.py file:

mkdir flask-docker-render
cd flask-docker-render
touch app.py

Note: At any time, you can view or make your own copy of the full code used in this article.

Now, let's go ahead and install Flask and Celery in our project:

pip install Flask gunicorn celery python-dotenv

To update your requirements.txt file with the newly-installed packages, run:

pip freeze > requirements.txt

You should see Flask and Celery appear in your requirements file (along with some other associated packages):

amqp==5.3.1
async-timeout==5.0.1
billiard==4.2.1
blinker==1.9.0
celery==5.5.1
click==8.1.8
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
Flask==3.1.0
gunicorn==23.0.0
itsdangerous==2.2.0
Jinja2==3.1.6
kombu==5.5.2
MarkupSafe==3.0.2
packaging==24.2
prompt_toolkit==3.0.50
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
six==1.17.0
tzdata==2025.2
vine==5.1.0
wcwidth==0.2.13
Werkzeug==3.1.3

Also, create config.py in your root and add:

class Config:
    DEBUG = False
    DEVELOPMENT = False
    CSRF_ENABLED = True

class ProductionConfig(Config):
    pass

class DevelopmentConfig(Config):
  DEBUG = True
  DEVELOPMENT = True

Now add this code to your app.py file:

# app.py

import os
from flask import Flask, jsonify
from tasks import generate_report
from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)

env_config = os.getenv("PROD_APP_SETTINGS", "config.DevelopmentConfig")
app.config.from_object(env_config)

@app.route('/start-task/')
def start_task():
    print("📬 /start-task was called!")
    task = generate_report.delay()
    return jsonify({"task_id": task.id, "status": "started"}), 202

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

In app.py, we build a very minimal Flask app that creates a /start-task/ route and imports the generate_report function from tasks.py. Let's add that below:

# tasks.py

import time
import os
from celery import Celery
from dotenv import load_dotenv
load_dotenv()

broker_url = os.environ.get("CELERY_BROKER_URL", "amqp://guest:guest@rabbitmq:5672//")
celery_app = Celery('tasks', broker=broker_url)

@celery_app.task
def generate_report():
    time.sleep(5)
    return "Report complete!"

In tasks.py, we initialize a Celery app and simulate a slow background task by using Python's built-in time module to add a 10-second delay. By separating the core Flask app from the Celery worker code, we ensure that our project is more modular and maintainable.

Note: We have also supplied the environment variable CELERY_BROKER_URL, which will come in handy in the next deployment section. (For now, in local development mode, the code just grabs the default local rabbitmq broker url).

Next, we will create a Dockerfile in the project root:

FROM python:3.10-alpine

WORKDIR /app

RUN apk add --no-cache gcc musl-dev linux-headers

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

COPY . .

EXPOSE 5000

RUN find . -name "*.pyc" -delete

CMD ["gunicorn", "-b", "0.0.0.0:5000", "app:app"]

The Dockerfile provides all the instructions needed to run a virtual container on our local machine. Let's break down each instruction here:

FROM python:3.10-alpine: We specify the Python version number and tell Docker we want the lightweight Alpine Linux version for our Linux container.
WORKDIR /app: Set the working directory to /app.
RUN apk add --no-cache gcc musl-dev linux-headers: Using the apk Linux Alpine package manager, we install the system-level build tools needed to compile our Python packages.
COPY requirements.txt requirements.txt: Copy the requirements.txt from the source local project to the container.
RUN pip install -r requirements.txt: Install all the packages listed in requirements.txt inside the container.
COPY ..: copy everything in your local project to the container (excluding anything in .dockerignore).
EXPOSE 5000: Tell Docker to listen on port 5000.
Delete stale .pyc files that are not needed and can cause problems for later deployments.
Tell Docker to run the Flask app using the production-grade Gunicorn web server and use app.py as the application entry point.

We also need to create a special yaml file for Docker called compose.yaml in the project root:

services:
  web:
    build: .
    ports:
      - "8000:5000"
    depends_on:
      - rabbitmq

  worker:
    build: .
    depends_on:
      - rabbitmq
    command: celery -A tasks.celery_app worker --loglevel=info

  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"
      - "15672:15672" # Management UI

In compose.yaml, we have a single configuration file to manage several services required for our project. In this case, the Flask app triggers the tasks (mapping the default Flask 5000 port to a custom 8000 port), a Celery worker that works in the background to complete the tasks, and the RabbitMQ server, which acts as the "broker" between both services. The rabbitmq section downloads the official RabbitMQ image from Docker Hub (once) and stores it for future use.

Lastly, we also need a separate Dockerfile.worker for our main Celery command to make sure that Celery runs correctly when we deploy to Render later in this tutorial (this file doesn't have any impact on our local Docker setup). We need this additional file as, unfortunately, Render does not currently support compose.yaml.

In the next deployment section, we will also need Render's own render.yaml file to ensure everything works smoothly.

# Dockerfile.worker

FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["celery", "-A", "tasks.celery_app", "worker", "--loglevel=info"]

The Dockerfile.worker is very similar to the Dockerfile for our Flask web service. The key difference here is the CMD (startup command). This tells Celery to start a worker service and look in tasks.py for the celery_app object.

With those five key files ready (app.py, tasks.py, Dockerfile, Dockerfile.worker, and compose.yaml) we can now build and start Flask, Celery, and RabbitMQ all with one Docker command:

docker compose up

The first time you run this command, you will see output that shows that RabbitMQ has been downloaded from the official Docker Hub image to the Docker container on your local machine. Any subsequent times you run the command, you will just see the boot-up output for Flask, Celery, and RabbitMQ.

Go to localhost:8000/start-task in your browser. If everything works correctly, you should see something very similar to the JSON below:

{
  "status": "started",
  "task_id": "14361d15-3a8e-4f73-8ed0-7f7fefc36efd"
}

Note: Your task_id will differ, as it is randomly generated each time.

To verify that the delayed Celery task is working, when you hit the start-task endpoint, you should see the following in your terminal logs:

[2025-04-15 16:00:16,306: INFO/MainProcess] Task tasks.generate_report[b7f6f260-5087-4893-b6ea-110888f620a6] received

Then, 10 seconds later, the task will show as successfully completed in the terminal:

[2025-04-15 16:00:26,331: INFO/ForkPoolWorker-2] Task tasks.generate_report[b7f6f260-5087-4893-b6ea-110888f620a6] succeeded in 10.017975573999138s: 'Report complete!'

Well done for making it this far! Now that we have everything working correctly on our local machine, it's time to deploy our project to Render!

Deploy Your Python Flask Project to Render

When deploying to Render, first create a render.yaml file:

services:
  - type: web
    name: flask-web
    env: docker
    plan: starter
    dockerfilePath: ./Dockerfile
    dockerContext: .
    envVars:
      - key: PROD_APP_SETTINGS
        value: config.ProductionConfig
      - key: CELERY_BROKER_URL
        sync: false

  - type: worker
    name: celery-worker
    env: docker
    plan: starter
    dockerfilePath: ./Dockerfile.worker
    dockerContext: .
    envVars:
      - key: CELERY_BROKER_URL
        sync: false

The render.yaml file acts as a blueprint to define all the infrastructure needed to deploy our app (in fact, Render actually calls this "Blueprints" in its UI). This means that if you have multiple services to deploy, everything is explicitly defined in the code. You don't have to add and configure each service in the UI. Also, when you set up everything via Render's Blueprint approach, you just have to push your main branch to GitHub, and it will trigger an automatic deploy for all your services.

Here's a quick explanation of everything that's happening in the render.yaml above. We define two services: one is a web service (our Flask app) and the second is the background Celery worker. Both are using Docker as their environment and are on the Render starter plan ($7 per month). One important thing to note is the two different Docker files used: the web service uses the plain Dockerfile, while the Celery worker uses Dockerfile.worker (both in the project root).

Lastly, the environment variables are pre-populated with a key and value for the web service. Since config.ProductionConfig is not sensitive information (but just used to automatically rotate between local and production environments), we have included this in the code. But the CELERY_BROKER_URL production environment variable should never be hardcoded. So we just provide the key for this, tell Render not to sync, and later we will manually enter the value in the Render UI.

Create CloudAMQP Instance

Now that we have our render.yaml file ready, let's sign up for an account at CloudAMQP, a managed cloud hosting service for RabbitMQ.

There are a few simple steps we need to follow in the CloudAMQP UI:

Click on Create New Instance and select the free Loyal Lemming plan.
Select your region and data center (AWS is fine), and your instance is created!
Click on your new instance to access the dashboard.
In your dashboard overview, look for the AMQP details and copy the AMQP URL. It will appear partly redacted in the dashboard, so make sure you copy the whole thing! This URL is crucial: we will need it to create our Render blueprint in the next section.

Create Render Blueprint

Go to Render and create a new account (if you haven't done so already).
Go to the Blueprints section and then click on + New Blueprint Instance.
You should see a list of your GitHub repositories: select the one you want to connect as a Blueprint.
In the next screen, name your Blueprint, then under Specified configurations, add your AMQP URL as the value associated with the CELERY_BROKER_URL key for both the web service and the background worker.
Click Deploy Blueprint. This will start the sync/deploy process.

Lastly, enter your new Render-generated URL (and don't forget the /start-task endpoint). So the URL should look something like this:

https://[YOUR-RENDER-URL].onrender.com/start-task/

If everything works correctly, you will see the same behavior as when you run the project locally. After hitting the start-task endpoint in your browser, go to Logs in your celery-worker dashboard. You should see something very similar to the following output:

[2025-04-24 15:31:34,645: INFO/MainProcess] Task tasks.generate_report[87ca895f-debc-4d07-bdde-a3408faa8c7e] received
[2025-04-24 15:31:44,657: INFO/ForkPoolWorker-16] Task tasks.generate_report[87ca895f-debc-4d07-bdde-a3408faa8c7e] succeeded in 10.010491968001588s: 'Report complete!'

Congratulations! You've made it to the end of this tutorial!

Wrapping Up

In this post, we ran through setting up and deploying a Flask app to Render using Docker.

First, we installed Docker on Mac (in the most painless way possible). Then, we created the local Docker version of our Flask app with Dockerfile, Dockerfile.worker, and compose.yaml files. Lastly, we prepared our app for deployment to Render with render.yaml, created a Render Blueprint, and a new CloudAMQP instance.

If you would like to take things further and integrate your Render-deployed app with AppSignal, check out AppSignal's Render integration docs.

You can also check out the docs to use the AppSignal standalone agent as a Docker image.

Happy coding!

How the Application and Request Contexts Work in Python Flask

Roy Tomeij — 2025-07-23T00:00:00+00:00

If you have spent some time developing Flask applications, you have probably encountered terms like request, session, current_app, and g. You might even use them daily. But have you ever stopped to think about how Flask makes these seemingly global objects available exactly when you need them, especially in a multi-threaded web server environment? Well, the magic lies in Flask's context system.

In this article, you will learn what contexts are in Flask and how to use them with practical examples.

Let’s dive in!

What Are Contexts in Flask?

In web frameworks, a context refers to a set of information relevant to the current state of processing. Think of it as a temporary workspace that holds all the tools and data needed for a specific task.

Why does Flask bother with this? Imagine a busy web server handling multiple user requests simultaneously. If objects like the current request details were truly global variables, different threads handling different requests could overwrite each other's data, leading to chaos!

Flask uses contexts to manage access to objects, such as the application instance and incoming request data, in a way that is thread-safe. Each thread gets its own version of the context, ensuring data isolation.

Flask primarily uses two types of contexts, which often work together:

Application Context: Deals with application-level data.
Request Context: Deals with data specific to a single incoming HTTP request.

Let's break down each one.

The Application Context: The App's Workspace

The application context is Flask's way of keeping track of application-level information that is not tied to a specific user request. Its main job is to make sure certain objects are accessible when needed, primarily the application instance itself.

When an application context is active, Flask makes available the following key objects:

current_app: This is a proxy object that points to the Flask application instance handling the current activity. It allows your blueprints, extensions, and other modules to access the app configuration, registered routes, etc., without needing you to pass the app object around explicitly.
g: It stands for 'global', though its scope is context-bound. Think of it as a temporary scratchpad. It is a namespace object where you can store arbitrary data during the life of an application context. A common use case is storing a database connection or the logged-in user details once per request, making them easily accessible elsewhere in your code during that same request handling cycle. However, consider that it only holds data for the duration of a single request. So, if you need to store user data that persists between a request, you must use other objects.

Flask automatically activates an application context before handling a request and deactivates it afterward. This usually happens hand-in-hand with the request context. You can also manually push an application context using the method push(). This is essential when you need access to current_app or g outside of a typical request cycle, like in background jobs, cron tasks, or Flask CLI commands.

The Request Context: Handling Individual Requests

While the application context deals with the app, the request context is all about a single incoming HTTP request from a client. It holds everything Flask needs to know to process that specific interaction.

The key objects available when a request context is active are:

request: This is probably the most frequently used context object. It is a proxy representing the incoming HTTP request. Through it, you can access form data (request.form), query parameters (request.args), uploaded files (request.files), headers (request.headers), and much more. It is your window into what the client is asking for.
session: This object allows you to store user-specific information across multiple requests. Flask's default session implementation uses cryptographically signed cookies. When you store something in session (e.g., session["user_id"] = user.id), Flask serializes it, signs it, and sends it back to the client as a cookie. On subsequent requests, Flask reads the cookie, verifies the signature, and makes the data available again via the session object. It is perfect for remembering who a user is between page loads.

Flask automatically pushes a request context right when it starts processing an incoming HTTP request and closes it when the response is generated and sent. Pushing a request context implicitly pushes an application context if one is not already active. This makes sense as you can not handle a request without knowing which application it belongs to. You generally do not need to manage the request context manually unless you are doing advanced testing or working with WebSockets outside the standard request-response flow.

Practical Examples

Understanding Flask contexts conceptually is one thing, but seeing them in action (and seeing things break when they're missing) really drives the point home. This section provides code examples to demonstrate:

How current_app, g, request, and session work within a standard request.
What happens when you try to access these objects outside a request context.
How to manually create contexts (app_context and test_request_context) to make these objects available when needed outside the normal request flow.

Prerequisites

To replicate this tutorial using Flask, you need to have at least Python 3.6 or newer installed on your machine.

Repository Structure and Requirements

Suppose you call the main folder of your project flask_contexts. At the end of this step, the folder will have the following structure:

flask_contexts/
   ├── venv/
   ├── app.py
   └── app_no_context.py

Where:

venv contains the virtual environment.
app.py contains the code that manages the contexts.
app_no_context.py contains the code that simulates a call outside the contexts.

You can create the venv virtual environment directory like so:

python -m venv venv

To activate it on Windows, run:

venv\Scripts\activate

Or on macOS and Linux, execute:

source venv/bin/activate

In the activated virtual environment, install the dependencies with:

pip install flask

Wonderful! You now have what you need to create a Flask application using contexts.

Step 1: Basic Setup and App Initialization

As a first step, you need to set the application up in the app.py file:

import time
import secrets
from flask import Flask, request, session, g, current_app, jsonify

# Application Setup
app = Flask(__name__)
app.secret_key = secrets.token_hex(16)

In this part of the code:

app = Flask(__name__): Creates an instance of the Flask application, where __name__ helps Flask determine the root path for resources.
app.secret_key = secrets.token_hex(16): Sets a secret key that generates a random 32-character hexadecimal string, suitable for a development environment.

Step 2: Define a Function That Needs Context

Define a Python function that attempts to access context-bound objects (current_app, g, request, and session). This function is designed to work correctly if called when the necessary contexts are active, but it will raise a RuntimeError if called when they are not:

def function_potentially_outside_context():
    """
    Tries to access context-bound objects.
    This will FAIL if called outside an active context.
    """
    try:
        # These require an active Application Context
        app_name = current_app.name
        g.some_value = "Set value in g"

        # These require an active Request Context (which includes App Context)
        method = request.method
        session_val = session.get("example", "Not Set")

        return (f"SUCCESS (Inside Context): App='{app_name}', Method='{method}', "
                f"Session='{session_val}', g.some_value='{g.some_value}'")

    except RuntimeError as e:
        # This is the typical error when accessing context objects outside a context
        return f"FAILED: Caught RuntimeError: {e}"
    except Exception as e:
        # Catch any other unexpected errors
        return f"FAILED: Caught unexpected Exception: {e}"

In this snippet, the try-except block manages the context-sensitive operations. If any of current_app, g, request, or session are accessed when their corresponding context is not active, Flask raises a RuntimeError.

Step 3: Set a Standard Route

Create a standard Flask route. When a user visits the root URL, Flask automatically sets up both the Application and Request contexts before executing the index function. Inside this function, you can freely use current_app, g, request, and session.

@app.route("/")
def index():
    """
    Standard route where Flask handles contexts automatically.
    Demonstrates 'g' reset and 'session' persistence.
    """
    print("\n---> Entering index route handler")

    # Check if 'g' has data from a previous request
    g_before = getattr(g, "request_timestamp", "Not Set Before This Request")
    print(f"Value of g.request_timestamp before setting: {g_before}")

    # Set a value in 'g' for this specific request
    g.request_timestamp = time.time()
    print(f"Set g.request_timestamp: {g.request_timestamp}")

    # Increment a session counter
    visit_count = session.get("visit_count", 0) + 1
    session["visit_count"] = visit_count
    print(f"Updated session['visit_count']: {visit_count}")

    # Access other context objects
    app_name = current_app.name
    req_method = request.method
    print(f"Accessed current_app.name: {app_name}")
    print(f"Accessed request.method: {req_method}")

    # Simulate work and use 'g' within the same request
    time.sleep(0.05)
    duration = time.time() - g.request_timestamp
    print(f"Request processing duration (using g): {duration:.4f}s")

    print("<--- Exiting index route handler")
    return jsonify({
        "message": "Inside standard request - contexts handled automatically.",
        "app_name": app_name,
        "request_method": req_method,
        "g_timestamp_set_in_this_request": g.request_timestamp,
        "session_visit_count": visit_count,
        "processing_duration_seconds": f"{duration:.4f}"
    })

This route does the following:

getattr(): Safely checks if g has the attribute request_timestamp from this request's context setup. This shows g is fresh for each request.
time.time(): Stores the current time in g. This value is only available during this specific request.
session.get() + 1: Retrieves visit_count from the session (defaulting to 0 if not found) and increments it at each new visit.
session["visit_count"] = visit_count: Stores the new count back in the session. This data will persist across requests for the same user.
current_app.name, request.method: Provide direct access works because Flask sets up the contexts.
jsonify(): Converts the Python dictionary into a JSON response for the browser.

Step 4: Call the Helper Function Within a Request

Create another route that calls the helper function from Step 2. Because this call happens inside a route handler where Flask has already set up the contexts, the helper function will succeed:

@app.route("/call-function-directly")
def call_function_directly():
    """
    Calls the helper function directly from within a request.
    Contexts are already active, so it should succeed.
    """
    print("\n---> Entering call_function_directly route handler")
    # Since we are inside a request handler, contexts are active.
    result = function_potentially_outside_context() # Call the helper
    print(f"Result from helper: {result}")
    print("<--- Exiting call_function_directly route handler")
    return jsonify({"test": "Calling function from within a route", "result": result})

In this route, result = function_potentially_outside_context() calls the function defined in Step 2. Since Flask ensures contexts are active here, the try-except block inside the function succeeds.

Step 5: Simulate the "No Context" Problem

This step highlights when you encounter problems. Create a route that simulates what happens if you try to call function_potentially_outside_context from outside Flask's request handling (like in a separate script or background task). The route itself can not truly demonstrate the failure (because it is a request), but this will be discussed later.

@app.route("/call-function-no-context")
def call_function_no_context_trigger():
    """
    Simulates calling the function where contexts might be missing.
    This route primarily serves to show the function's output format.
    """
    print("\n---> Entering call_function_no_context_trigger route handler")
    # Simulate the call outside of a request context
    result = "Simulated Failure: Imagine 'RuntimeError: Working outside of application context.' was raised here."
    print("<--- Exiting call_function_no_context_trigger route handler")
    return jsonify({
        "test": "Simulating call without context (via route)",
        "result": result,
        "note": "To see real failure, run function_potentially_outside_context() from a plain script."
        })

This function will be invoked in app_no_context.py next to show the actual simulation outside the contexts.

Step 6: Running the Application

To run the app.py application, write the following:

if __name__ == "__main__":
    print("\nStep 8: Starting Flask development server...")
    print("Access the examples at:")
    print("- http://127.0.0.1:5000/")
    print("- http://127.0.0.1:5000/call-function-directly")
    print("- http://127.0.0.1:5000/call-function-no-context")
    print("\nRun 'python test_no_context.py' in another terminal to see the context error.")
    app.run(debug=True, host='0.0.0.0', port=5000)

Perfect! Now your app.py file is ready to be tested!

Step 7: Manage the Request Outside the Context

To manage the request outside the context, write the following in the app_no_context.py file:

from app import app, function_potentially_outside_context

print("Attempting to call function outside Flask's request handling...")
result = function_potentially_outside_context()
print(f"Result: {result}")

Now you are ready to also test the app_no_context.py file.

Step 8: Testing

As a first test, try the main application by writing:

python app.py

This is what you will see when you access the URL http://127.0.0.1:5000/:

The browser shows that you have been successful, and each time you refresh the page, the session_visit_count will increase, as expected. Even the timestamp will be different, as it is managed by the main route.

This is the result when you access the URL http://127.0.0.1:5000/call-function-directly:

Because the function is called from within a route where Flask has already set up the contexts, the result is a SUCCESS message.

The interesting part is to simulate the request externally from the contexts. To do so, open a new terminal and type:

python app_no_context.py

You will obtain the following result on the CLI:

This demonstrates the necessity of a context, as the function has been called outside of it, on purpose.

Step 9: Put it All Together

Below is what the app.py file should now contain:

import time
import secrets
from flask import Flask, request, session, g, current_app, jsonify

# Application Setup
app = Flask(__name__)
app.secret_key = secrets.token_hex(16)


def function_potentially_outside_context():
    """
    Tries to access context-bound objects.
    This will FAIL if called outside an active context.
    """
    try:
        # These require an active Application Context
        app_name = current_app.name
        g.some_value = "Set value in g"

        # These require an active Request Context (which includes App Context)
        method = request.method
        session_val = session.get("example", "Not Set")

        return (f"SUCCESS (Inside Context): App='{app_name}', Method='{method}', "
                f"Session='{session_val}', g.some_value='{g.some_value}'")

    except RuntimeError as e:
        # This is the typical error when accessing context objects outside a context
        return f"FAILED: Caught RuntimeError: {e}"
    except Exception as e:
        # Catch any other unexpected errors
        return f"FAILED: Caught unexpected Exception: {e}"


@app.route("/")
def index():
    """
    Standard route where Flask handles contexts automatically.
    Demonstrates 'g' reset and 'session' persistence.
    """
    print("\n---> Entering index route handler")

    # Check if 'g' has data from a *previous* request (it shouldn't)
    g_before = getattr(g, "request_timestamp", "Not Set Before This Request")
    print(f"Value of g.request_timestamp before setting: {g_before}")

    # Set a value in 'g' for this specific request
    g.request_timestamp = time.time()
    print(f"Set g.request_timestamp: {g.request_timestamp}")

    # Increment a session counter
    visit_count = session.get("visit_count", 0) + 1
    session["visit_count"] = visit_count
    print(f"Updated session['visit_count']: {visit_count}")

    # Access other context objects
    app_name = current_app.name
    req_method = request.method
    print(f"Accessed current_app.name: {app_name}")
    print(f"Accessed request.method: {req_method}")

    # Simulate work and use 'g' within the same request
    time.sleep(0.05)
    duration = time.time() - g.request_timestamp
    print(f"Request processing duration (using g): {duration:.4f}s")

    print("<--- Exiting index route handler")
    return jsonify({
        "message": "Inside standard request - contexts handled automatically.",
        "app_name": app_name,
        "request_method": req_method,
        "g_timestamp_set_in_this_request": g.request_timestamp,
        "session_visit_count": visit_count,
        "processing_duration_seconds": f"{duration:.4f}"
    })

@app.route("/call-function-directly")
def call_function_directly():
    """
    Calls the helper function directly from within a request.
    Contexts are already active, so it should succeed.
    """
    print("\n---> Entering call_function_directly route handler")
    # Since you are inside a request handler, contexts are active.
    result = function_potentially_outside_context()
    print("<--- Exiting call_function_directly route handler")
    return jsonify({"test": "Calling function from within a route", "result": result})

@app.route("/call-function-no-context")
def call_function_no_context_trigger():
    """
    Simulates calling the function where contexts might be missing.
    This route primarily serves to show the function's output format.
    """
    print("\n---> Entering call_function_no_context_trigger route handler")
    # Simulate the call outside of a request context
    result = "Simulated Failure: Imagine 'RuntimeError: Working outside of application context.' was raised here."
    print("<--- Exiting call_function_no_context_trigger route handler")
    return jsonify({
        "test": "Simulating call without context (via route)",
        "result": result,
        "note": "To see real failure, run function_potentially_outside_context() from a plain script."
        })

if __name__ == "__main__":
    print("Starting Flask development server...")
    print("Access the examples at:")
    print("- http://127.0.0.1:5000/")
    print("- http://127.0.0.1:5000/call-function-directly")
    print("- http://127.0.0.1:5000/call-function-no-context")
    app.run(debug=True, host='0.0.0.0', port=5000)

Discussion About the "No Context" Simulation

The app_no_context.py example shows that things break, but not when you could encounter it naturally, because you simulated it—somehow forcing it.

So, when can the app_no_context.py scenario happen in real life?

The automatic context setup only happens during a live web request. There are many situations where you might want to run parts of your Flask application's code outside of a direct web request. This is where you need to manually create contexts.

Here are common real-world examples where you would be "outside" a request and need contexts:

Background jobs/task queues: A user uploads a video (web request). Your view function adds a task to a queue (like Celery) to process the video later. While the trigger to add a task to the queue might come from an HTTP request handled by Flask, the actual execution of the task happens later in a completely separate worker process. This worker process is not the same process as the Flask web server handling requests. It simply pulls a job description from a queue (like Redis or RabbitMQ) and runs the associated Python code. Since this worker process is not directly handling an incoming HTTP request, Flask doesn't automatically set up any contexts for it.
Custom Flask CLI commands: You write a command like flask create-admin --email admin@example.com to add an admin user directly from your terminal. In this case, you are executing a Python script via the Flask command-line interface runner. There is no incoming HTTP request involved. You are interacting directly with the script on the command line. Because there's no web request being processed, Flask's automatic request-response context setup mechanism does not run.
Scheduled tasks: You have a Python script run by cron every night to clean up old user sessions or generate reports. Similarly to CLI commands, this execution happens independently of any web server or incoming HTTP request. Since Flask is not involved in handling an HTTP request at that moment, it doesn't automatically create the contexts.

Wrapping Up

In this article, you learned that Flask's context system is a fundamental mechanism that ensures application and request data are managed correctly and safely, especially in concurrent environments. By distinguishing between the Application Context and the Request Context, Flask offers a robust way to handle state.

As demonstrated, attempting to access context-bound objects outside of an active context leads to a RuntimeError. Recognizing when these situations arise allows developers to manually push the necessary contexts.

Mastering Flask contexts allows you to write more reliable, maintainable, and testable applications, enabling you to leverage Flask's full potential beyond simple request handling.

Happy testing!

How to Avoid N+1 Queries in Django Python

Roy Tomeij — 2025-07-09T00:00:00+00:00

Django is a powerful web framework that simplifies how developers interact with databases through its Object-Relational Mapping (ORM) system. However, even with its benefits, it’s easy to fall into performance pitfalls such as the N+1 query problem.

In this article, we’ll explore what N+1 queries are, why they can be an issue for your application, and how to mitigate them using Django’s best practices.

Let's dive in!

Understanding the N+1 Query Problem in Django

Before diving into solutions, it’s essential to understand the fundamentals of this performance bottleneck.

Imagine you’re creating a Django application that needs to display a list of authors and their corresponding books. You might write a simple query to retrieve all the authors, and then iterate over each one to pull their books. At first glance, this appears straightforward. However, if your application makes a separate database query for each author’s books, you might end up issuing many more queries than you intended: a classic symptom of the N+1 query problem.

In Django, N+1 queries occur when one initial query (the “1”) is followed by additional queries (the “N”) for each object retrieved. This scenario usually surfaces when code is structured in a way that iterates over many objects and lazily retrieves related data. The result can be significant performance degradation when interacting with a database. For example, if you have 100 authors and you access each author’s books in a loop, Django could run 101 queries in total, rather than a single, optimized query. This pattern not only increases latency, but also puts unnecessary load on your database server.

The impact of N+1 queries on database performance can be substantial. As the number of objects grows, so does the number of queries, leading to a linear increase in database calls. This kind of inefficiency can slow down page load times and strain your application, especially when your database scales or when operating in a high-traffic environment. This issue brings to light the importance of querying data as efficiently as possible and leveraging Django’s abilities to ensure optimized performance.

How N+1 Queries Arise in Django

The N+1 problem typically appears in scenarios where developers do not explicitly optimize the way related objects are queried. Two common patterns that lead to this pitfall are:

Accessing related fields in templates: Django’s ORM performs lazy loading, which means that when you reference a related field in a template, Django may execute a separate query for each object accessed. For example, if you loop through a list of articles in a template and display each article’s author, Django might issue an extra query for each article to fetch the corresponding author details. This situation leads to N+1 queries because one query retrieves the articles, and each article then triggers another query for its associated author.
Iterating over related objects in views: A second common pattern occurs in view logic, particularly when iterating over a queryset and accessing related objects without preloading. Consider a scenario where you retrieve a list of authors and then, within a loop in your view, access each author’s set of books. If you don't optimize your query, Django will execute a separate query for each author to fetch their books. This repetition causes query numbers to proliferate as the number of authors grows, resulting in an N+1 query scenario.

How to Avoid N+1 Queries

Now that we've gone through the theory, it is time to provide a step-by-step tutorial. This section first shows how N+1 queries look, then two ways of solving the issue using Django.

We will use Django to retrieve a list of authors and related books from a blog.

Requirements

To replicate the tutorial, you must have Python 3.10.1 or higher installed on your machine.

Prerequisites and Dependencies

Suppose you call the main folder of your project django_queries/. In the beginning, the folder should look like this:

django_queries/
    └── venv/

Where venv/ contains the virtual environment. You can create the venv/ virtual environment like so:

python3 -m venv venv

To activate it on Windows, run:

venv\Scripts\activate

On macOS and Linux, execute:

source venv/bin/activate

In the activated virtual environment, install the dependencies:

pip install django

Step 1: Create a New Django Project

Start a new project like so:

django-admin startproject nplus1_project

This will automatically create subfolders with Python files. Go to the nplus1_project/ subfolder:

cd nplus1_project

Start a new Django application called blog:

python manage.py startapp blog

The structure of your repository should now look like this:

nplus1_project/
    ├── blog/
    │   ├── migrations/
    │       └── __init__.py
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── models.py
    │   ├── tests.py
    │   └── views.py
    ├── nplus1_project/
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    └── manage.py

Step 2: Define Models

The N+1 query problem typically arises when querying related models. So, let’s create two models: Author and Post, where each Post is written by an Author.

Define the models in the blog/models.py file, like so:

from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=100)

    def __str__(self):
        return self.name

class Post(models.Model):
    title = models.CharField(max_length=200)
    content = models.TextField()
    author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name='posts')

    def __str__(self):
        return self.title

Apply the migrations to Django by running:

python manage.py makemigrations
python manage.py migrate

Now you've created the basis of this tutorial.

Step 3: Populate the Database

The next step is to populate the database with some data. Open the Django shell:

python manage.py shell

Add the following data to the database:

from blog.models import Author, Post

# Create authors
author1 = Author.objects.create(name="Author 1")
author2 = Author.objects.create(name="Author 2")

# Create posts
Post.objects.create(title="Post 1", content="Content 1", author=author1)
Post.objects.create(title="Post 2", content="Content 2", author=author1)
Post.objects.create(title="Post 3", content="Content 3", author=author2)
Post.objects.create(title="Post 4", content="Content 4", author=author2)

If you encounter an error at this stage that says something like "NameError: name 'Post' is not defined", it means that Django doesn't recognize your blog application or its models. To fix this, go into the settings.py file inside the nplus1_project directory, search for the INSTALLED_APPS list and add your blog app name, like so:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'blog',  # Add your app here
]

Save the settings.py file and apply the migrations again:

python manage.py makemigrations
python manage.py migrate

Step 4: Create a View with the N+1 Query Problem

In this step, we'll actually create the N+1 query issue on purpose to help you understand how to visualize it. To do so, open blog/views.py and create a view:

from django.shortcuts import render
from blog.models import Post

def post_list(request):
    # This will cause the N+1 query problem
    posts = Post.objects.all()
    for post in posts:
        print(post.author.name)  # Accessing the related author for each post

    return render(request, 'blog/post_list.html', {'posts': posts})

In this code, the post_list() view queries all the Post objects. Then, for each Post, it accesses the related Author object, causing a separate query for each Post, thus creating the N+1 query issue.

Step 5: Create a URL for the View

Create a file called urls.py in the folder blog/ and define the URL for the post_list view:

from django.urls import path
from . import views

urlpatterns = [
    path('', views.post_list, name='post_list'),
]

Also, set the nplus1_project/urls.py file like this:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('blog.urls')),
]

Step 6: Create a Template

To visualize how data is retrieved and displayed by the UI, you need to create a template. Create a directory for the templates in blog/templates/blog/. Inside it, create a file named post_list.html with the following code:

<!DOCTYPE html>
<html>
  <head>
    <title>Post List</title>
  </head>
  <body>
    <h1>Posts</h1>
    <ul>
      {% for post in posts %}
      <li>{{ post.title }} by {{ post.author.name }}</li>
      {% endfor %}
    </ul>
  </body>
</html>

The final repository structure looks like this:

django_queries/
│    ├── blog/
│    │   ├── migrations/
│    │   ├── templates/
│    │   │   └── blog/
│    │   │       └── post_list.html
│    │   ├── __init__.py
│    │   ├── admin.py
│    │   ├── apps.py
│    │   ├── models.py
│    │   ├── tests.py
│    │   ├── urls.py
│    │   └── views.py
│    ├── nplus1_project/
│    │   ├── __init__.py
│    │   ├── asgi.py
│    │   ├── settings.py
│    │   ├── urls.py
│    │   └── wsgi.py
│    ├── db.sqlite3
│    └── manage.py
└── venv/

Step 7: Run The Application

In nplus1_project/settings.py, enable the possibility to log SQL queries via the console by adding the following:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
            'handlers': ['console'],
        },
    },
}

Now, run the server:

python manage.py runserver

Go to http://127.0.0.1:8000/ and see the server running. The listed data is displayed like this:

However, what matters to us is what is happening under the hood. Here is the expected output you will see on the CLI:

This is an N+1 query issue, as you can observe:

One query to fetch all Post objects.
Additional queries for each Post to fetch its related Author object.

Now, let's see how to solve the issue.

Step 9: Fix the N+1 Query Problem With `select_related()`

One way to solve the N+1 query is to use the select_related() method. To do so, modify the blog/views.py file like so:

def post_list(request):
    # Use select_related to avoid the N+1 query problem
    posts = Post.objects.select_related('author')
    for post in posts:
        print(post.author.name)  # No additional queries are executed here

    return render(request, 'blog/post_list.html', {'posts': posts})

Run the server again. This is the expected result:

In this case, the query fetches all Post objects, joins the Author table using the author_id foreign key, and retrieves all the necessary fields in a single query.

Hooray! You fixed the problem!

Step 10: Fix the N+1 Query Problem With `prefetch_related()`

Another way to solve this issue is to use the prefetch_related() method. Modify the blog/views.py file like so:

def post_list(request):
    # Use prefetch_related to avoid the N+1 query problem
    posts = Post.objects.prefetch_related('author')
    for post in posts:
        print(post.author.name)  # Accessing the related author for each post

    return render(request, 'blog/post_list.html', {'posts': posts})

After running the server again, this is the expected result:

And again, the data is retrieved in a single query!

When to Use `prefetch_related()` or `select_related()`

You may be wondering when to use select_related() over prefetch_related(). This depends on the type of relationship between your models and your specific use case:

select_related() is used for single-valued relationships, such as ForeignKey or OneToOneField. It performs a single SQL query with a JOIN to fetch the related data.
prefetch_related() is used for multi-valued relationships, such as ManyToManyField or reverse ForeignKey relationships. It performs two separate queries and uses Python to match the related objects.

Here is a summary table:

Feature	`select_related()`	`prefetch_related()`
Use Case	Single-valued relationships (`ForeignKey`, `OneToOneField`)	Multi-valued relationships (`ManyToManyField`, reverse `ForeignKey`)
Number of Queries	1 query with `JOIN`	2 queries (main query + prefetch query)
Performance	Faster for single-valued relationships	Better for large datasets or multi-valued relationships
Database Load	Uses `JOIN`, which can be expensive for large datasets	Separate queries, which can be more efficient for large datasets

Level Up Django Monitoring with AppSignal

If you are a Django user and want to level up your monitoring capabilities, consider trying AppSignal's integration for Django. To spot N+1 queries, AppSignal has an instrumentation events feature that detects if the same event is repeated more than once in succession. Such events are marked as potential N+1 events and shown in the Performance tab of your dashboard, under Issue list:

And that's it!

Wrapping Up

In this article, you learned about N+1 queries in Django. First, we explored how N+1 queries come to exist in your application, before diving into how to solve them using prefetch_related() and select_related().

Happy coding!

How to Use MongoDB in Python Flask

Roy Tomeij — 2025-07-02T00:00:00+00:00

When developing software applications, data storage is a key concern. The reality is that your first concern should be the data model you choose, which in turn affects how you store data. Generally speaking, this means deciding between SQL and NoSQL databases.

In this article, you will learn how to use MongoDB, a popular NoSQL database, in a Flask application. First, you will learn why MongoDB is a good choice, and then we will implement a practical hands-on project using MongoDB in Flask.

Why Use MongoDB in a Python Flask Application?

MongoDB is an open-source document-oriented database management system (DBMS) classified as NoSQL, as it stores data in JSON-like documents (the so-called BSON).

MongoDB was designed with scalability and high availability in mind, making it suitable for large-scale web applications. It also provides automatic sharding capabilities, allowing users to distribute their data across multiple servers without having to worry about complex configurations or manual partitioning processes.

So, given its characteristics, here are some reasons why you might want to consider using MongoDB in your Flask applications:

Flexibility with data modeling: Unlike the classical relational databases, MongoDB allows you to define flexible schemas for your collections, given its NoSQL nature. This means that it allows you to work with unstructured and semi-structured data.
Scalability: MongoDB supports horizontal scaling through sharding, which enables distributing data across multiple servers while maintaining data consistency (all the shards form a sharded cluster, which acts as a single logical database). This allows you to handle increased loads to high degrees by increasing read/write throughput and storage capacity.
Easy integration: MongoDB integrates seamlessly with many programming languages and frameworks, including Python. In the case of Flask, for example, there are several libraries that make the integration process smooth, like pymongo, flask-pymongo, and Flask-MongoAlchemy.
Real-time applications: MongoDB provides the ability to manage real-time data streams efficiently by integrating seamlessly with other technologies such as Apache Kafka.

How to Use MongoDB in Flask: A Step-by-step Guide

Now that you know why you should use MongoDB with Flask, let's dive into a step-by-step tutorial on integrating it into your application. We'll implement basic CRUD operations for a "User" model.

Prerequisites

Before starting, ensure that you have installed the following:

Python version 3.x
pip package manager for Python
virtualenv module for creating isolated environments in Python projects
Any recent version of MongoDB

NOTE: As a prerequisite, you also need to create a new database named my_database in MongoDB: it will be used later in this tutorial. To do so:

Launch MongoDB (the command depends on your Operating System and version of MongoDB) after you have installed it.
Type use my_database to create a new database named my_database.
Insert data into a collection to make it usable. You can type something like: db.my_collection.insertOne({ name: "John Doe", age: 30 }).

Step 1: Define Project Structure

Supposing you call your main folder my_flask_mongo_app. Create the following structure:

my_flask_mongo_app/
│
├── app/
│   ├── __init__.py
│   ├── models.py
│   ├── routes.py
│   └── config.py
│
├── venv/
│
└── run.py

NOTE: The venv directory will be created when you activate the virtual environment later.

Step 2: Create The Virtual Environment and Install the Needed Libraries

Now, create the virtual environment.

Linux/macOS::

python3 -m venv venv
source venv/bin/activate

Windows:

python -m venv venv
venv\Scripts\activate

Then install the needed libraries via pip:

pip install flask flask_pymongo

Step 3: Configure The Flask Application

In the app/ folder, compile the Python files as follows.

In config.py type:

import os

class Config:
    SECRET_KEY = os.urandom(24)
    MONGO_URI = "mongodb://localhost:27017/my_database"

This file defines the configuration for the Flask application. In particular:

MONGODB_URI specifies the database name, host, and port for connecting to MongoDB. In this case, you'll be using the local MongoDB server running on port 27017 and accessing my_database.
SECRET_KEY secures sessions and other security-related features in Flask.

In __init__.py write the following code:

from flask import Flask
from flask_pymongo import PyMongo
from .config import Config

# create PyMongo instance at module level
mongo = PyMongo()

def create_app():
    app = Flask(__name__)
    app.config.from_object(Config)

    # Initialize mongo with the app
    mongo.init_app(app)

    from .routes import main
    app.register_blueprint(main)

    return app

This file initializes the Flask application by:

Creating a Flask instance and loading the configuration from Config.
Initializing PyMongo, which allows Flask to interact with MongoDB.
Importing and registering the main blueprint from routes.py, which contains the application's routes.

Step 4: Define MongoDB Models

In the models.py file, write the following:

from . import mongo

class User:
    @staticmethod
    def create(username, email):
        user = {
            'username': username,
            'email': email
        }
        mongo.db.users.insert_one(user)
        return user

    @staticmethod
    def get_by_username(username):
        return mongo.db.users.find_one({'username': username})

    @staticmethod
    def update(username, new_email):
        mongo.db.users.update_one(
            {'username': username},
            {'$set': {'email': new_email}}
        )

    @staticmethod
    def delete(username):
        mongo.db.users.delete_one({'username': username})

This file defines the 'User' model with static methods for CRUD operations:

The create() method inserts a new user document into the collection of users.
The get_by_username() method retrieves a user document by username.
The update() method updates a user's email.
The delete() method deletes a user document by username.

Step 5: Implement Routes and CRUD Operations

Use the routes.py file to create the actual CRUD operations via routes in Flask:

from flask import Blueprint, request, jsonify
from .models import User

main = Blueprint('main', __name__)

@main.route('/users', methods=['POST'])
def create_user():
    data = request.json
    user = User.create(data['username'], data['email'])
    return jsonify(user), 201

@main.route('/users/<username>', methods=['GET'])
def get_user(username):
    user = User.get_by_username(username)
    if user:
        return jsonify(user), 200
    return jsonify({'error': 'User not found'}), 404

@main.route('/users/<username>', methods=['PUT'])
def update_user(username):
    data = request.json
    User.update(username, data['email'])
    return jsonify({'message': 'User updated'}), 200

@main.route('/users/<username>', methods=['DELETE'])
def delete_user(username):
    User.delete(username)
    return jsonify({'message': 'User deleted'}), 200

This code defines the API endpoints (routes) for the Flask application. In particular, each route corresponds to a CRUD operation:

POST /users: Creates a new user.
GET /users/<username>: Retrieves a user by username.
PUT /users/<username>: Updates a user's email.
DELETE /users/<username>: Deletes a user.

Step 7: Create and Run the Main Application File

To create the main application, write the following code into the run.py file:

from app import create_app

app = create_app()

if __name__ == '__main__':
    app.run(debug=True)

This file creates the Flask app and MongoDB connection using the create_app() function from app/__init__.py.

Then, run the application by typing:

python3 run.py

While the application runs, you can test the API endpoints using tools like Postman or curl.

For example, to create a new user, send a POST request to http://localhost:5000/users with the following JSON payload:

{
  "username": "john_doe",
  "email": "johndoe@example.com"
}

You can do it via curl like so:

curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"username": "john_doe", "email": "john@example.com"}' \
     http://localhost:5000/users

That's all! Now you know the basics of how to use MongoDB in Flask.

Wrapping Up

In this article, we learned how to use MongoDB in Flask. First, we explored why and when you should use MongoDB. Then, we offered a step-by-step guide on using MongoDB in Flask that supports and implements basic CRUD operations for a "User" model.

Oh, and by the way, did you know that AppSignal provides MongoDB and Flask integrations? Give them a try for a superior monitoring experience with AppSignal!

Happy coding!

Flask or Django: Which One Best Fits Your Python Project?

Roy Tomeij — 2025-06-25T00:00:00+00:00

Choosing the right framework can significantly impact the success of a dev project.

If you've chosen to use Python for your project's backend, you might need to decide between Flask and Django, the most popular web frameworks in the Python ecosystem.

This article will help you determine which of the two best suits your specific needs by examining the key factors to consider.

Before diving into the technicalities, let's briefly introduce both Flask and Django.

Introducing Flask and Django for Python

We'll begin by examining what Flask has to offer.

Introducing Flask

Flask is an open source, Python-based web microframework developed by Armin Ronacher in 2011. It comes with a lot of tools, technologies, and libraries. It offers form validation and other extensions for object-relational mapping, open authentication, file uploads, and more.

Its key features and benefits are:

Lightweight and flexible: As a microframework, Flask provides the essentials for web development without additional components by default. Its lightweight nature allows developers to pick and choose the libraries and tools that best fit their project requirements, providing maximum flexibility.
Jinja2 templating: Flask uses Jinja2, a powerful and flexible templating engine, for rendering dynamic HTML that supports template inheritance, macros, and filters. This streamlines the process of creating reusable and maintainable templates.
Extensions support: Flask supports the introduction of third-party extensions to add functionality to web applications. Among the many extensions available are Flask-SQLAlchemy, which adds support for working with databases, and Flask-Login which provides user session management.
Development server and debugger: Flask includes a built-in development server and debugger that provides error messages and interactive debugging tools. The debugger can execute code in the context of a stack trace, helping developers diagnose and fix issues efficiently during development.

Introducing Django

Django is an open source, Python-based web framework that was released in 2005 by Adrian Holovaty and Simon Willison. It provides full-stack development configurations, such as template layouts, requests and troubleshooting, cookies, form validation, unit tests, table settings, and other functionality used by developers to create dynamic web applications.

Its offers:

A comprehensive framework: As a high-level framework, Django includes a wide range of built-in features like authentication, Object-Relational Mapping (ORM), and an admin interface. This "batteries-included" approach allows developers to focus on writing application-specific code rather than reinventing the wheel for common tasks.
Object-Relational Mapping (ORM): Django’s ORM enables developers to interact with their database using Python code instead of SQL. This abstraction layer supports multiple database backends (including PostgreSQL, MySQL, and SQLite), simplifying database migrations and schema management.
Admin interface: Django automatically generates an admin interface for defined models in an application. This feature is highly customizable and provides a quick way to manage and administer site content without additional coding.
Security: Django has robust built-in security features, such as protection against SQL injection, cross-site scripting (XSS), cross-site request forgery (CSRF), and clickjacking. It also encourages developers to adopt security best practices and provides mechanisms like password hashing and user authentication.

While Flask and Django are both Python-based and open source, the main difference between them is that Flask is a microframework, whereas Django is a full-featured framework. This is an important differentiator.

Now, let's dive into the key factors to consider when deciding which to choose for your web development project.

Comparing Flask and Django

Let's make a head-to-head comparison between Flask and Django.

Learning Curve

An important factor to take into account is the learning curve needed for each framework.

Flask is often celebrated for its simplicity and ease of use, making it an excellent choice for beginners and those who prefer a minimalistic approach.

Flask is simple and easy to use because it is a micro-framework. This means that it provides the basic tools needed to build a web application and leaves developers to decide which additional components to use. Because it's lightweight, developers can get started quickly.

In contrast to Flask, Django follows a "batteries-included" philosophy, providing a comprehensive set of features. As we've mentioned, these include Object-Relational Mapping for database interactions, a powerful admin interface, authentication mechanisms, and a templating engine. While this richness of features can be overwhelming for beginners, it significantly reduces the amount of third-party code and configurations needed for more complex applications.

To summarize, if you’re a beginner familiar with Python, it’s easy to get your head around Flask’s minimalist structure to start your project. Django is a more complex web solution that requires extensive expertise for sophisticated applications, so it is more suited for experts.

Development Time

Flask can significantly reduce development time, but you usually need to build most parts of your website from scratch, so it can be quite labor-intensive. If you're an experienced developer, you might find your development time slowed down by Flask's limited built-in features.

On the other hand, Django supports rapid development, especially when you're working under tight deadlines. This is because it comes with many built-in features that reduce the amount of code you need to write. For example, developers can create a minimum viable product (MVP) faster with Django than with Flask. This makes Django a great choice for starting large websites quickly.

While Flask also allows for quick MVP development, Django has a clear advantage when it comes to the frontend. Django's built-in template engine, in fact, speeds up development compared to Flask's. Additionally, Django has many standard libraries that help developers build common functionalities and solve typical development problems.

Database Management

Django and Flask offer distinct approaches to database management.

Django, as already mentioned, includes a robust ORM that supports a variety of relational databases such as SQLite, PostgreSQL, MySQL, and Oracle, making it an excellent choice for projects that utilize these databases. Its ORM simplifies the process of generating and managing database migrations and facilitates the creation of forms and views, which is ideal for web applications performing CRUD operations.

However, consider that, for complex queries or projects using NoSQL databases, Django’s ORM can be limiting, although third-party projects like Django MongoDB Engine or Djongo can provide some support.

On the other hand, Flask provides developers with complete freedom in choosing how to handle data storage. This flexibility allows for the use of a wide range of libraries and extensions, such as the already mentioned Flask-SQLAlchemy for relational databases or Flask-PyMongo for MongoDB. While this flexibility can result in a steeper learning curve and a higher risk of errors, it also enables developers to select the ORM or ODM (Object Document Mapper) that best suits their specific needs.

So, if your project relies heavily on a relational database, Django’s integrated ORM will likely make development smoother and more straightforward. Instead, if your project involves non-relational databases or requires custom data handling solutions, Flask’s flexibility might be more advantageous (despite the increased complexity and learning required).

Security

Flask includes some basic security features, like Cross-Site Request Forgery (CSRF) protection, but it relies heavily on third-party extensions for additional security measures. While Flask's lower coding requirements can reduce the risk of certain cyber threats, the reliance on third-party extensions means that your application's security depends on the strength of these plugins. This places a significant responsibility on your development team to regularly assess and update third-party libraries and extensions to ensure security remains robust.

Django, on the other hand, comes with a wide range of built-in security features, including user password hashing, CSRF tokens, and authentication and authorization modules. These built-in tools help prevent common security mistakes and allow developers to run a comprehensive security checklist before deploying their applications. Additionally, the Django development team is proactive in identifying and reporting security vulnerabilities, ensuring security issues are addressed promptly.

As a result, Django is generally easier to secure from the start, and to maintain throughout the lifecycle of your application.

Performance

When it comes to performance, Flask’s lightweight nature often translates to faster response times for simple applications. Its minimalistic design ensures there is little overhead, making it an excellent choice for high-performance applications where speed is important.

You can consider the following general rules of thumb:

Flask: The response times are often faster for simple endpoints due to its minimalistic design and lower overhead. Flask's lightweight nature means there’s less processing involved for each request compared to Django.
Django: While Django can be slightly slower for very simple endpoints, it offers more built-in functionality and a robust framework for larger applications. The added features and middleware in Django, however, might introduce some additional latency.

In practice, the actual response time differences between the two might be small, but Flask’s minimalism could lead to lower latency in scenarios where every millisecond counts. Django’s performance advantages come into play with more complex applications due to its powerful feature set and the optimizations available for larger projects.

Scalability

Both Flask and Django are capable of handling scalable applications, but they approach scalability differently.

Flask’s micro-framework nature makes it easier to design a distributed architecture where components can be scaled independently. This can be particularly useful in microservice architectures, where different parts of an application are developed and deployed separately.

Django’s scalability often comes from its integrated features that support large-scale applications. The framework includes tools for managing large databases, handling high traffic, and implementing advanced caching mechanisms. Django’s scalability is also enhanced by its compatibility with asynchronous task queues such as Celery, allowing for efficient background processing.

Community and Ecosystem

In today's tech world, a community around a product is a great asset for companies because it provides support as well as integration.

Flask has a vibrant and active community, which is a significant advantage for developers. The community's contributions have resulted in a rich ecosystem of third-party libraries and extensions that significantly extend Flask’s capabilities.

Popular forums, GitHub repositories, and resources like Stack Overflow provide ample support for developers encountering issues or seeking advice.

Here are some key resources for Flask:

Django’s community is one of its greatest strengths. The framework has been around since 2005, leading to a mature and well-established ecosystem. This longevity means that many common web development problems have already been solved, and solutions are readily available.

The Django community is known for its robust third-party packages, which can be found in the Django Packages repository. Additionally, the Django Software Foundation supports the development of the framework and organizes events like DjangoCon, fostering a strong sense of community and continuous improvement.

Key resources for Django include:

Flask vs. Django Comparison Table

Here's a table that summarizes all the key aspects of Flask and Django we've covered in this article, to help you make a better and more informed decision on which to choose:

Factor	Flask	Django
Learning Curve	Simple and easy to use: ideal for beginners	Rich feature set, steeper learning curve: suitable for experts
Development Time	Requires building parts from scratch. Can be labor-intensive	Rapid development with many built-in features
Database Management	Flexible, supports various ORMs and ODMs	Integrated ORM, supports multiple relational databases
Security	Basic built-in security, relies on third-party extensions	Highly secure by default with extensive built-in features
Performance	Faster for simple applications due to minimalistic design	Slightly slower for simple endpoints, but optimized for larger applications
Scalability	Easy to design distributed architecture, good for microservices	Integrated tools for large-scale applications and high traffic management
Community and Ecosystem	Vibrant and active community, rich ecosystem of third-party libraries	Mature and well-established community, robust third-party packages

Wrapping Up

Whether you choose Flask or Django depends on the specific needs of your project, your expertise as a developer (or your team's capabilities), and your long-term goals.

As we've covered, Flask’s simplicity and flexibility make it an excellent choice for smaller projects or applications requiring significant customization.

Django, with its comprehensive feature set and integrated components, is ideal for complex projects that can benefit from built-in tools and a robust framework structure.

By carefully evaluating your project's requirements and considering the strengths and weaknesses of each framework, you can make an informed decision that will set the foundation for a robust Python project.

Happy coding!

Ways to Optimize Your Code in Python

Roy Tomeij — 2025-05-28T00:00:00+00:00

By optimizing Python code, you improve performance, reduce resource consumption, and enhance scalability. While Python is known for its simplicity and readability, these characteristics can sometimes come at the cost of efficiency.

In this article, we'll explore four ways to optimize your Python project and improve performance.

First, we'll look at how best to use data structures.

Efficient Use of Python Data Structures

We'll use some of the most well-known Python data structures to optimize our code.

Lists Vs. Tuples

Lists and tuples are probably the most basic and well-known data structures in Python. They serve different purposes, so they have different performance characteristics:

Lists are mutable, which means they can be modified after creation.
Tuples, instead, are immutable.

Before diving deep into why there are performance differences, let's write a code sample that creates a list and a tuple of 5 numbers.

import timeit

# Calculate creation time
list_test = timeit.timeit(stmt="[1, 2, 3, 4, 5]", number=1000000)
tuple_test = timeit.timeit(stmt="(1, 2, 3, 4, 5)", number=1000000)

# Print results
print(f"List creation: {list_test: .3} seconds")
print(f"Tuple creation: {tuple_test: .3} seconds")

This results in:

List creation:  0.135 seconds
Tuple creation:  0.0207 seconds

To calculate performance differences, we use the timeit module like so:

The stmt parameter defines the code snippet we want to evaluate. So, in the case of the list_test variable, it evaluates a list of five numbers; in tuple_test, it evaluates a tuple of five numbers.
The number parameter specifies how many times the stmt parameter must be executed. In both cases, we run it 100,0000 times, meaning the code creates the list and the tuple 100,0000 times.

As the example shows, tuples are way faster than lists. Let's dig into why:

Memory allocation:
- Due to their immutability, tuples are stored in a fixed-size block of memory. The size of this block is determined when the tuple is created, and it doesn’t change. This fixed size makes tuple memory allocation fast.
- Lists, on the other hand, need to support dynamic resizing. This means they often allocate extra space to accommodate potential growth without requiring frequent reallocations.
Internal structure:
- The internal structure of a tuple consists essentially of a continuous block of memory with a fixed layout. This layout includes the elements themselves and some metadata (like size), but since tuples are immutable, the structure remains simple.
- Lists have a more complex internal structure to manage their mutability. They need to keep track of their current size and allocated capacity, and must handle changes in size dynamically.
Caching and optimization:
- Python can use various optimizations for tuples, such as caching, because their immutability guarantees that they won’t change after creation. These optimizations reduce the need for repeated memory allocation and speed up creation.
- While Python does optimize list operations, the potential for lists to change means that optimization is limited compared to tuples.

Dictionaries and Sets Vs. Lists in Python

In Python, dictionaries and sets are data structures that allow for fast lookups. When you want to check if an item is in a set or find a value associated with a key in a dictionary, these operations typically take constant time; this is denoted as O(1) in "Big O notation".

Given their structure, using dictionaries and sets can significantly improve performance when you need to frequently check for the existence of an item or access elements by a key.

Let's show this with a code snippet. For example, suppose we create a dictionary, a set, and a list with 100,0000 numbers. We want to look for the number 999,999 and then work out how long it takes using these three different data structures:

import time

# Create a large set, dictionary, and list
large_set = {i for i in range(1000000)}
large_dict = {i: str(i) for i in range(1000000)}
large_list = [i for i in range(1000000)]

# define element to lookup
element = 999999

# Timing set lookup
start_time = time.time()
found = element in large_set
end_time = time.time()
print(f"Set lookup took: {end_time - start_time:.8f} seconds")

# Timing dictionary lookup
start_time = time.time()
found = element in large_dict
end_time = time.time()
print(f"Dictionary lookup took: {end_time - start_time:.8f} seconds")

# Timing list lookup
start_time = time.time()
found = element in large_list
end_time = time.time()
print(f"List lookup took: {end_time - start_time:.8f} seconds")

The result is:

Set lookup took: 0.00000000 seconds
Dictionary lookup took: 0.00000000 seconds
List lookup took: 0.00771618 seconds

So, basically, the time needed to search for the element 999,999 is (almost) 0 seconds for the set and the dictionary.

Of course, if we want to compare the sets and the dictionary, we'll find that the set provides better performance, as we may expect:

import timeit

# Calculate timing performance for dictionary and set
dict_test = timeit.timeit(stmt="'a' in {'a': 1, 'b': 2, 'c': 3}", number=1000000)
set_test = timeit.timeit(stmt="1 in {1, 2, 3, 4, 5}", number=1000000)

# Print results
print(f"Dictionary lookup: {dict_test: .3} seconds")
print(f"Set lookup: {set_test: .3} seconds")

This results in:

Dictionary lookup:  0.0821 seconds
Set lookup:  0.0212 seconds

So, how do dictionaries and sets achieve O(1)?

Well, dictionaries and sets use a data structure called a hash table. Here's a simplified explanation of how it works:

Hashing: When you add a key to a dictionary or an item to a set, Python computes a hash value (a fixed-size integer) from the key or item. This hash value determines where the data is stored in memory.
Direct access: With the hash value, Python can directly access the location where the data is stored without searching the entire data structure.

So it's very fast to check if an item exists in a set or dictionary, which is useful for operations that require frequent existence checks.

Choosing the Right Data Structure

Choosing the appropriate data structure based on the specific needs of your application leads to significant performance gains.

If you need to store data and you're sure it won't change over time, definitely use tuples to optimize your code.

When you need to frequently look for elements, prefer sets and dictionaries over lists or tuples.

Global Variables, Encapsulation, and Namespace

In Python, scope determines the visibility and lifetime of a variable in a program. Variables can have different scopes:

Local Scope: This refers to variables defined within a function. They are only accessible inside that function.
Global Scope: Variables defined at the top level of a script or module. They are accessible throughout the module.
Class/instance Scope: Variables defined within a class, including class attributes and instance attributes.

This section describes code optimization by avoiding global variables, using class encapsulation, and managing a namespace correctly.

Avoiding Global Variables

Local variables are faster to access compared to global variables, primarily due to the way Python manages variable scopes and lookups.

In particular, Python uses the Local, Enclosing, Global, Built-in (LEGB) rule to resolve variable names:

Local: Names defined within a function.
Enclosing: Names in the local scopes of any enclosing functions.
Global: Names at the top level of the module or script.
Built-in: Preassigned names in the Python built-in namespace.

When accessing a variable, Python starts searching from the innermost scope (the local one). Since the local scope is limited to the function’s context, it contains fewer variables, making the search process quicker. On the contrary, global scope encompasses all top-level names in the module, resulting in a potentially larger search space.

Here's an example to show the difference in performance when using global vs. local variables:

import time

# Local variable test
def local_test():
    a = 0
    for var_1 in range(1000000):
        a += 1

# Global variable test
b = 0
def global_test():
    global b
    for var_2 in range(1000000):
        b += 1

start_time = time.time()
local_test()
local_time = time.time() - start_time
print(f"Local variable test:{local_time: .3} seconds")

start_time = time.time()
global_test()
global_time = time.time() - start_time
print(f"Global variable test: {global_time: .3} seconds")

And the result is:

Local variable test: 0.0441 seconds
Global variable test:  0.0685 seconds

So, whenever possible, prefer using local variables to global ones.

Encapsulation

Encapsulating variables within functions and classes can improve performance by reducing the scope and limiting the number of variables the interpreter needs to track.

Let's see the difference in performance between using encapsulation and not using it:

import timeit

# Class without encapsulation
class Rectangle:
    '''This class creates a rectangle without encapsulation'''
    def __init__(self, width, height):
        self.width = width
        self.height = height

    def area(self):
        return self.width * self.height

    def perimeter(self):
        return 2 * (self.width + self.height)

# Class with encapsulation
class EncapsulatedRectangle:
    '''This class creates a rectangle with encapsulation'''
    def __init__(self, width, height):
        self._width = width
        self._height = height

    def get_width(self):
        return self._width

    def set_width(self, width):
        self._width = width

    def get_height(self):
        return self._height

    def set_height(self, height):
        self._height = height

    def area(self):
        return self._width * self._height

    def perimeter(self):
        return 2 * (self._width + self._height)

# Create instances of both classes
rect = Rectangle(10, 20)
enc_rect = EncapsulatedRectangle(10, 20)

# Define the test functions
def test_rect_area():
    return rect.area()

def test_enc_rect_area():
    return enc_rect.area()

def test_rect_perimeter():
    return rect.perimeter()

def test_enc_rect_perimeter():
    return enc_rect.perimeter()

# Time the functions using timeit
iterations = 5000000

rect_area_time = timeit.timeit(test_rect_area, number=iterations)
enc_rect_area_time = timeit.timeit(test_enc_rect_area, number=iterations)
rect_perimeter_time = timeit.timeit(test_rect_perimeter, number=iterations)
enc_rect_perimeter_time = timeit.timeit(test_enc_rect_perimeter, number=iterations)

# Print results
print(f"Rectangle (no encapsulation) area time: {rect_area_time:.4f} seconds")
print(f"Encapsulated Rectangle area time: {enc_rect_area_time:.4f} seconds")
print(f"Rectangle (no encapsulation) perimeter time: {rect_perimeter_time:.4f} seconds")
print(f"Encapsulated Rectangle perimeter time: {enc_rect_perimeter_time:.4f} seconds")

The result is:

Rectangle (no encapsulation) area time: 2.1583 seconds
Encapsulated Rectangle area time: 2.2764 seconds
Rectangle (no encapsulation) perimeter time: 2.5185 seconds
Encapsulated Rectangle perimeter time: 2.4265 seconds

Encapsulation can significantly improve performance in larger applications with numerous variables. By keeping variables local to functions and classes, you reduce the interpreter's workload since it has fewer variables to manage. This leads to a faster execution time, especially in complex programs with many functions and classes.

These are the key benefits of encapsulation:

Reduced scope: By limiting the scope of variables to the smallest necessary context, the interpreter has fewer variables to track, which leads to faster execution.
Memory management: Local variables are automatically deallocated when a function exits, which helps with efficient memory use.
Avoiding naming conflicts: Encapsulation prevents variable name clashes, making code easily maintainable and less error-prone.

So, you'd better use encapsulation to improve performance when creating classes. Also, note that encapsulation provides controlled access and modification of attributes, protecting data from outside modifications.

Correct Namespace Management

Minimizing global namespace pollution leads to better performance. The main idea, in this case, is to use modules and packages to organize your code and keep the global namespace clean. In other words, instead of using too many global variables and creating too many functions or classes, create modules and import them.

Here's a general example of inefficient namespace management:

global_var_1 = "..."

def first_function():
    '''A function'''
    ....

def second_function():
    '''Another function'''
    ...

def main_function():
    '''The main function of the program'''
    ...

The main idea is to change this to something like:

from functions.function_1 import *
from functions.function_2 import *

def main_function():
    '''The main function of the program'''
    ...

To do so, you have to modularize your functions so that the structure of your folders becomes something like:

main_folder/
|__ main.py
|
|__ functions/
        |__ function_1.py
        |__ function_2.py

NOTE: In this case, function_1.py and function_2.py can be bigger than first_function() and second_function().

So, whenever possible, create modules and packages from your code. This helps with:

Performance: The code is more efficient on the machine side.
Readability: Shorter code is generally more easily readable than longer code. It's better to have small connected programs than a big program.
Reuse: Any module or package you create can be used in other programs, helping you save time in the future.

Utilize List Comprehensions and Generator Expressions

This section describes how code performance can be improved through list comprehension and generators.

List Comprehension

List comprehension is a fast and concise way to create a new list using the power of loops and statements with one line of code.

Let's see the difference in performance first:

import timeit

# Define the code snippets as functions
def loop_code():
    '''This function creates a new list out of classic for loop'''
    squares = []
    for x in range(10):
        squares.append(x**2)

def comprehension_code():
    ''''This function created a new list out of a list comprehension'''
    squares = [x**2 for x in range(10)]

# Measure execution time
loop_test = timeit.timeit(loop_code, number=1000000)
comprehension_test = timeit.timeit(comprehension_code, number=1000000)

print(f"Loop: {loop_test: .4} seconds")
print(f"List comprehension: {comprehension_test: .4} seconds")

This leads to:

Loop:  2.642 seconds
List comprehension:  2.444 seconds

List comprehension is more performance-friendly than standard for-loops because of:

Reduced overhead: List comprehensions are implemented in C within the Python interpreter, making them faster (as lower-level optimizations aren't accessible in a standard Python for-loop).
No method calls: In a traditional for-loop, the append() method is called repeatedly, which adds some overhead. List comprehensions avoid this by constructing the list in a single expression.
Local scope: Variables defined within a list comprehension are scoped more tightly than variables defined in a for-loop. This reduces the potential for variable conflicts and can sometimes make garbage collection more efficient.

So, whenever possible, always prefer using list comprehension to create a new list. This enhances performance and code readability.

Generator Expressions

Generator expressions in Python provide a concise way to create generators without using a separate generator function with the yield() method. They are similar to list comprehensions — the key difference being that they produce values one at a time and only when needed, which makes them more memory-efficient for large data sets.

Let's see how generator expressions can improve performance:

import timeit

# Define the size of the iterable
n = 1000000

# Generator expression
gen_expr = (i for i in range(n))

# List comprehension
list_comp = [i for i in range(n)]

# Measure the time taken by the generator expression
gen_time = timeit.timeit('sum((i for i in range(n)))', globals=globals(), number=10)

# Measure the time taken by the list comprehension
list_time = timeit.timeit('sum([i for i in range(n)])', globals=globals(), number=10)

print(f"Generator expression took {gen_time:.4f} seconds")
print(f"List comprehension took {list_time:.4f} seconds")

This results in:

Generator expression took 1.4823 seconds
List comprehension took 1.9738 seconds

Generator expressions are more efficient than list comprehension due to:

Lazy evaluation: Generator expressions generate items on the fly. This means that they do not compute all items at once, which is memory efficient.
Memory efficiency: Since values are produced one at a time, generator expressions use less memory compared to list comprehensions, especially for large datasets.

If you need to store values and use them only when needed, always prefer generators for good performance.

Leveraging Built-in Functions and Libraries

This section describes how optimizing code using built-in libraries and functions improves the performance of your machine.

Standard Library Efficiency

Python’s standard library functions are often implemented in C and optimized for speed. Using these functions leads to significant performance improvements, due to:

Lower-level operations: C operates closer to the hardware level compared to Python, providing more efficient memory and CPU usage.
Optimized algorithms: Experienced developers highly optimize standard library functions to perform common tasks efficiently.
Reduced overhead: Invoking a function implemented in C avoids the overhead associated with Python's dynamic typing and interpreted execution.

Suppose we sort a list with a lot of numbers. The following example compares performance when creating a custom function versus using a built-in one:

import time

# Sorting via custom function
def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(0, n-i-1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[j]

arr = [i for i in range(10000, 0, -1)]
start_time = time.time()

bubble_sort(arr)

end_time = time.time()
print(f"Bubble sort took: {end_time - start_time} seconds")

# Sorting via built-in function
arr = [i for i in range(10000, 0, -1)]
start_time = time.time()

sorted(arr)

end_time = time.time()
print(f"Sorted function took: {end_time - start_time} seconds")

The performance results:

Bubble sort took: 25.067534685134888 seconds
Sorted function took: 0.0 seconds

The built-in sorted() function immediately performs the sorting operation. The custom function, on the other hand, takes nearly 30 seconds to complete its tasks.

Using Third-party Libraries

We can also use third-party libraries like NumPy (a library that brings the computational power of languages like C and Fortran to Python) and Pandas (a fast, powerful, flexible, and easy-to-use open source data analysis and manipulation tool, built on top of the Python programming language) for performance optimizations. These libraries are highly optimized for numerical computations and data manipulation, so, for performance reasons, it's always better to use them rather than to create a custom function.

Suppose we want to add up an array's elements. We can do so with a custom function or with the method np.sum() by Numpy:

import time
import numpy as np

def sum_array(arr):
    total = 0
    for num in arr:
        total += num
    return total

# Create a large array
large_array = list(range(1, 10000001))

# Measure time for custom function
start_time = time.time()
custom_sum = sum_array(large_array)
custom_duration = time.time() - start_time


# Convert the list to a NumPy array
large_array_np = np.array(large_array)

# Measure time for NumPy function
start_time = time.time()
numpy_sum = np.sum(large_array_np)
numpy_duration = time.time() - start_time

print(f"Duration with custom function: {custom_duration: .4} seconds")
print(f"Duration with Numpy: {numpy_duration: .4} seconds")

And here's the result:

Duration with custom function:  1.023 seconds
Duration with Numpy:  0.008745 seconds

The difference in performance is huge in this case!

So, remember: you don't need to reinvent the wheel. One of Python's superpowers is that it relies on a vast range of both standard and third-party libraries. You can always use them to save coding and computation time.

Wrapping Up

In this article, we've described four ways to optimize your Python code to improve your machine's performance (and save coding time). We hope you find these tips and tricks useful.

Happy coding!

Using JWTs in Python Flask REST Framework

Roy Tomeij — 2025-04-30T00:00:00+00:00

JSON Web Tokens (JWTs) secure communication between parties over the internet by authenticating users and transmitting information securely, without requiring a centralized storage system.

In this article, we'll explain what JWTs are and give a high-level overview of how they work. We'll also implement a JWT-based authentication system by creating a to-do list API using Flask.

So, whether you're a beginner or an experienced Python developer, this guide aims to enhance your understanding of JWTs and their practical application in Flask.

First, let's provide an overview of JWTs and explain how they work, at a high level.

JWTs: How They Work

JWTs are a compact and self-contained method for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed using a secret or a public/private key pair.

A JWT is composed of three parts:

Header: Contains metadata about the token, including the type of token and the hashing algorithm used to sign it. Typical cryptographic algorithms include HMAC with SHA-256 (HS256) and an RSA signature with SHA-256 (RS256). Here's an example of how this might look:
```
{
  "alg": "HS256",
  "typ": "JWT"
}
```
In this case, alg specifies the hashing algorithm used, while typ indicates the token type (which, of course, is "JWT"). Read about other commonly used header fields.
Payload: Contains the claims, which are statements about an entity (typically, the user) and additional data that can include information like the user ID, username, and roles. Here's an example of how the payload might look:
```
{
  "sub": "1234567890",
  "name": "John Doe",
  "admin": true,
  "iat": 1516239022
}
```
In this case, sub is the subject identifier and represents the subject of the token (e.g., the user ID); name is the name of the user; admin is a boolean indicating administrative privileges; iat means "issued at time" and represents when the token was issued (in the Unix timestamp).
Signature: The signature is used to verify the authenticity of the token and to ensure that it hasn't been tampered with. It is created by encoding the header and payload, concatenating them with a period, and then hashing them using the algorithm specified in the header along with a secret key.

So, when putting everything together, the structure of a JWT looks like this:

{Header in Base64URL}.{Payload in Base64URL}.{Signature}

When decoded, the JWT reveals the original JSON structures:

{
  "header": {
    "alg": "HS256",
    "typ": "JWT"
  },
  "payload": {
    "sub": "1234567890",
    "name": "John Doe",
    "admin": true,
    "iat": 1516239022
  },
  "signature": "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
}

Why JWTs Are Useful for Secure User Authentication in APIs

JWTs offer several benefits when securing user authentication in APIs, including:

Stateless sessions: JWTs eliminate the need to store session information on the server, as all the necessary data is contained within a token. This makes scaling your application easier.
Security: Because JWTs are signed, the server can verify the integrity and authenticity of the token. This ensures that the data hasn't been altered and comes from a trusted source. This also helps protect endpoints by ensuring that only authenticated users can access certain resources.
Efficiency: JWTs are compact, making them ideal for inclusion in HTTP headers and efficient for network transmission.
Flexibility: They can include a variety of claims, allowing you to convey user roles, permissions, and other metadata necessary for authorization.
Cross-domain support: JWTs can be used across different domains without issues, making them suitable for Single Sign-On (SSO) systems.
Simplified authentication flow: They streamline the authentication process, reducing the complexity of managing user sessions.

Setting Up the Flask Environment for a Flask Python App

Let's now dive into a practical example to show how to implement theory into practice.

But before that, let's first set up a Flask environment with all the necessary dependencies.

Installing Dependencies and Project Structure

First, make sure you have Python 3.6+ installed. Then, create a project directory like this one:

jwt_flask_todo/
├── app.py
└── venv/

Where:

app.py is the main Flask application file.
venv/ is the virtual environment where dependencies will be installed.

To create the virtual environment, navigate to the root folder of your project and execute the command below:

python3 -m venv venv

Then activate it:

Linux/MacOS:

source ./venv/bin/activate

Windows:

.\venv\Scripts\activate.bat

You can now install the libraries needed to replicate the Flask app:

pip install Flask Flask-JWT-Extended

Note that Flask-JWT-Extended is the library used in the following code examples to manage JWT tokens.

Implementing JWT Authentication in a Flask API

We'll start our to-do list API by implementing user registration and login functionality, integrating JWTs for authentication.

Building a User Registration System

Let's first create an endpoint to register users, storing their data in an in-memory dictionary for simplicity:

from flask import Flask, request, jsonify

app = Flask(__name__)

# In-memory 'database' of users
users = {}

@app.route('/register', methods=['POST'])
def register():
    username = request.json.get('username')
    password = request.json.get('password')

    # Input validation
    if not username or not password:
        return jsonify({"msg": "Missing username or password"}), 400

    # Check if user already exists
    if username in users:
        return jsonify({"msg": "User already exists"}), 400

    # Store user
    users[username] = password
    return jsonify({"msg": "User registered successfully"}), 201

if __name__ == '__main__':
    app.run(debug=True)

This endpoint does the following:

Data retrieval: Extracts a username and password from the JSON request body.
Validation: Checks if the username and password are provided.
User existence check: Ensures the username isn't already taken.
User storage: Adds the new user to the users dictionary.
Response: Returns a success message if everything works as expected.

Adding User Login with JWTs

Next, let's create a login endpoint that issues a JWT token when a user logs in successfully:

from flask_jwt_extended import JWTManager, create_access_token

# Configure the JWT secret key
app.config['JWT_SECRET_KEY'] = 'your_jwt_secret_key'
jwt = JWTManager(app)

@app.route('/login', methods=['POST'])
def login():
    username = request.json.get('username')
    password = request.json.get('password')

    # Input validation
    if not username or not password:
        return jsonify({"msg": "Missing username or password"}), 400

    # Authentication
    if users.get(username) != password:
        return jsonify({"msg": "Invalid credentials"}), 401

    # Create JWT
    access_token = create_access_token(identity=username)
    return jsonify(access_token=access_token), 200

This endpoint is similar to the user registration one. In addition, it does the following:

Authentication: Checks if the provided password matches the stored one.
Token creation: Generates an access token with the create_access_token method, embedding the username as the identity.

Protecting To-Do List API Endpoints with JWTs

Now that you have implemented a user registration and login, you can secure the endpoints so that only authenticated users can access them.

Apply the @jwt_required() decorator to the routes you want to protect. Use get_jwt_identity() to retrieve the current user's identity from the token:

from flask_jwt_extended import JWTManager, jwt_required, get_jwt_identity

# Initialize the JWTManager:
app = Flask(__name__)
app.config['JWT_SECRET_KEY'] = 'your_jwt_secret_key'
jwt = JWTManager(app)

# Secure the endpoint
@app.route('/protected', methods=['GET'])
@jwt_required()
def protected():
    # Access the identity of the current user w
    current_user = get_jwt_identity()
    return jsonify(logged_in_as=current_user), 200

So, here's what this code does:

@jwt_required() decorator: Ensures that the endpoint can only be accessed if a valid JWT is present in the request. If the JWT is missing or invalid, the request will be rejected.
get_jwt_identity() function: Inside the protected route, this function retrieves a current user's identity from the JWT. The identity is the value you set when creating the token (usually, the username or user ID).

Creating a To-Do List API

Let's now build a simple to-do list API where each user can manage their tasks. We'll create endpoints to manage the classical CRUD operations we'd expect to see in such an app.

Creating a Task

First, let's create a task:

@app.route('/tasks', methods=['POST'])
@jwt_required()
def add_task():
    current_user = get_jwt_identity()
    task = request.json.get('task')

    if not task:
        return jsonify({"msg": "Task content missing"}), 400

    # Initialize user's task list if it doesn't exist
    if current_user not in to_do_lists:
        to_do_lists[current_user] = []

    # Add task to user's task list
    to_do_lists[current_user].append(task)
    return jsonify({"msg": "Task added", "task": task}), 201

This endpoint does the following:

@jwt_required(): Ensures that only authenticated users can access the endpoint.
get_jwt_identity(): Retrieves the current user's username from the JWT.
Task retrieval: Gets the task content from the request JSON.
Validation: Checks if the task content is provided.
Task list initialization: Creates an empty list for the user if they don't have one.
Adding task: Appends the task to the user's list.
Response: Returns a confirmation message with the task added.

Viewing All Tasks

This endpoint fetches the task list for a current user and returns an empty list if none exists:

@app.route('/tasks', methods=['GET'])
@jwt_required()
def get_tasks():
    current_user = get_jwt_identity()
    tasks = to_do_lists.get(current_user, [])
    return jsonify({"tasks": tasks}), 200

Updating a Task

Now we'll update a task:

@app.route('/tasks/<int:task_id>', methods=['PUT'])
@jwt_required()
def update_task(task_id):
    current_user = get_jwt_identity()
    tasks = to_do_lists.get(current_user, [])

    if task_id < 0 or task_id >= len(tasks):
        return jsonify({"msg": "Task not found"}), 404

    updated_task = request.json.get('task')
    if not updated_task:
        return jsonify({"msg": "Task content missing"}), 400

    tasks[task_id] = updated_task
    return jsonify({"msg": "Task updated", "task": updated_task}), 200

In this endpoint:

task_id parameter: Represents the index of the task in the user's task list.
Index validation: Ensures the task ID is within valid bounds.
Updated task content: Retrieves the new task content from the request.
Updating task: Replaces the existing task with the updated content.
Response: Confirms the update with the new task.

Deleting a Task

Finally, this endpoint removes the task at the specified index:

@app.route('/tasks/<int:task_id>', methods=['DELETE'])
@jwt_required()
def delete_task(task_id):
    current_user = get_jwt_identity()
    tasks = to_do_lists.get(current_user, [])

    if task_id < 0 or task_id >= len(tasks):
        return jsonify({"msg": "Task not found"}), 404

    deleted_task = tasks.pop(task_id)
    return jsonify({"msg": "Task deleted", "task": deleted_task}), 200

For longer-lived sessions, you can allow users to refresh their tokens instead of having to log in again after every expiration.

Implementing Token Refresh for Longer Sessions

You can set up token refreshes for longer sessions in a few different ways:

Update the JWT configuration.
Modify the login endpoint to issue refresh tokens.
Create a new endpoint that manages token refreshes.

Let's take each approach in turn.

Updating JWT Configuration

JWTs expiration time can be managed like so:

from datetime import timedelta

app.config['JWT_ACCESS_TOKEN_EXPIRES'] = timedelta(minutes=15)  # Access tokens expire after 15 minutes
app.config['JWT_REFRESH_TOKEN_EXPIRES'] = timedelta(days=30)    # Refresh tokens expire after 30 days

Modifying the Login Endpoint to Issue Refresh Tokens

To use both access and refresh tokens, you can update the login endpoint as follows:

from flask_jwt_extended import create_refresh_token

@app.route('/login', methods=['POST'])
def login():
    # ... [existing code]

    # Authentication
    if users.get(username) != password:
        return jsonify({"msg": "Invalid credentials"}), 401

    # Create tokens
    access_token = create_access_token(identity=username)
    refresh_token = create_refresh_token(identity=username)
    return jsonify(access_token=access_token, refresh_token=refresh_token), 200

In this case, you can use the create_refresh_token() method to generate a refresh token for the user and prolong their session.

Adding Token Refresh Endpoint

As in the last case, you can create an endpoint to refresh access tokens using the refresh token:

from flask_jwt_extended import jwt_refresh_token_required

@app.route('/refresh', methods=['POST'])
@jwt_refresh_token_required
def refresh():
    current_user = get_jwt_identity()
    new_access_token = create_access_token(identity=current_user)
    return jsonify(access_token=new_access_token), 200

Here, though, the @jwt_refresh_token_required decorator ensures the endpoint is accessed with a valid refresh token. Then, the create_access_token() method creates a new access token to refresh the endpoint.

Logging Out and Managing User Sessions

Since JWTs are stateless, they cannot be invalidated server-side once issued. For this reason, you can implement token blocklisting to simulate a logout.

Let's see how.

Blocklisting Tokens

Revoked tokens can be maintained into a blocklist like so:

from flask_jwt_extended import get_raw_jwt

# Set to store blocklisted tokens
blocklisted_tokens = set()

@app.route('/logout', methods=['DELETE'])
@jwt_required()
def logout():
    jti = get_raw_jwt()['jti']
    blocklisted_tokens.add(jti)
    return jsonify({"msg": "Successfully logged out"}), 200

The get_raw_jwt() method retrieves the JWT data, including the jti (which is the JWT ID). Then, this endpoint adds the token's jti to the blocklisted_tokens set, simulating a logout. Once a token is inserted into the blocklist, any attempt to use it to log in will fail.

So, before processing any request, you could check if the token a user is trying to use has been revoked. This could be done like so:

from flask_jwt_extended import jwt_required, get_jwt_claims

@app.before_request
def check_revoked_token():
    jti = get_raw_jwt().get('jti')
    if jti in blocklisted_tokens:
        return jsonify({"msg": "Token revoked"}), 401

So, this piece of code can run before every request and checks if the token's jti is in the blocklist. If it is, it returns a response indicating that the token has been revoked.

Adding Permissions and Differentiating Between User Roles

To enhance security, you can manage control access to certain endpoints. This can be achieved, for example, through Role-Based Access Control (RBAC).

So, let's see how you can implement a simple RBAC system that differentiates between standard users and admins. But before that, bear in mind that in this article:

Standard users can only manage their own tasks.
Admin users have full access, including the ability to delete any task.

Modifying User Registration to Include Roles

First, you have to modify a registration endpoint to allow setting a user role. This can be done like so:

@app.route('/register', methods=['POST'])
def register():
    # Extract data from the request
    username = request.json.get('username')
    password = request.json.get('password')
    role = request.json.get('role', 'user')  # Default role is 'user'

    # Input validation
    if not username or not password:
        return jsonify({"msg": "Missing username or password"}), 400

    # Check if user already exists
    if username in users:
        return jsonify({"msg": "User already exists"}), 400

    # Store user with role
    users[username] = {'password': password, 'role': role}
    return jsonify({"msg": "User registered successfully"}), 201

Here's what's happening in this endpoint now:

Role assignment: Now there's the possibility of setting a role during registration. In particular, note that the default assigned value is user if no role is explicitly provided.
User data structure: The code stores user information as a dictionary containing the password and role. In other words, it stores the user with its own role.

Protecting Endpoints Based on Roles

To improve endpoint protection even further, you can protect based on users' roles.

Create a custom decorator that checks a user's role before allowing access to certain endpoints, like so:

from functools import wraps
from flask import abort

def role_required(required_role):
    def decorator(f):
        @wraps(f)
        @jwt_required()
        def wrapper(*args, **kwargs):
            claims = get_jwt()
            user_role = claims.get('role', 'user')
            if user_role != required_role:
                abort(403, description="Forbidden: Insufficient privileges")
            return f(*args, **kwargs)
        return wrapper
    return decorator

In this example, the role_required decorator checks if the user's role matches the required_role (the one needed to get access). If the user's role doesn't match, the endpoint aborts with a 403 Forbidden error message, meaning that the requester doesn't have the right privileges to authenticate the endpoint.

Finally, we'll take a brief look at setting permissions so that only admins can delete tasks.

Specific Operations: Only Admins Can Delete Tasks

There are cases where deleting something should only be allowed for administrators.

This can be implemented in the to-do list app we created by modifying the delete task endpoint, allowing only admins to delete tasks. Here's how:

from flask_jwt_extended import get_jwt

@app.route('/tasks/<int:task_id>', methods=['DELETE'])
@role_required('admin')
def delete_task(task_id):
    # Get all tasks (for all users)
    all_tasks = [task for tasks in to_do_lists.values() for task in tasks]

    # Check if task exists
    if task_id < 0 or task_id >= len(all_tasks):
        return jsonify({"msg": "Task not found"}), 404

    # Find and delete the task
    for user_tasks in to_do_lists.values():
        if task_id < len(user_tasks):
            deleted_task = user_tasks.pop(task_id)
            return jsonify({"msg": "Task deleted by admin", "task": deleted_task}), 200
        else:
            task_id -= len(user_tasks)

So now, the delete task endpoint uses the @role_required('admin') decorator to ensure only admins can access this endpoint. This means that only admins are allowed to delete any task from any user. Then, it confirms that the deletion is done.

And that's it!

Wrapping Up

By creating a to-do list API, we demonstrated how JWTs can secure user data and ensure that only authenticated users can access or modify their tasks.

We covered registration, login, task management, token refresh, and logout functionality, giving you the foundation to expand these concepts further!

Happy coding!

Instrumenting Redis in Python with AppSignal

Roy Tomeij — 2025-04-08T00:00:00+00:00

Using AppSignal, you can monitor the performance of your Redis calls and get insights into how your application uses Redis. The AppSignal Python agent supports instrumenting Redis calls out of the box, so you can start monitoring your usage with just a few lines of code.

In this post, we will cover how to instrument Redis in a Python application using AppSignal. We will build a simple URL shortener that stores data in a local Redis instance and uses AppSignal to monitor performance. Latencies, errors, and other metrics will be available in your dashboard in AppSignal, so you can keep an eye on how your application uses Redis.

Ready? Let’s get started!

Spin Up a Python Project

We will assume you have a Redis instance running locally and a basic understanding of Python. You can follow the instructions on the Redis website to install Redis on your machine. If you are using Windows, you can use the Windows Subsystem for Linux (WSL) to run Redis.

First, let’s check that the Redis service is on your machine by running the following command in your terminal:

> redis-cli ping
PONG
> lsof -i :6379

If PONG is the response and the lsof command doesn't output anything, then Redis is running on your machine.

Next, let’s create a new Python project:

> mkdir url-shortener
> cd url-shortener

Now, let’s build the requirements.txt file that will contain the dependencies we need for the project:

appsignal
opentelemetry-instrumentation-redis
opentelemetry-instrumentation-flask
redis
pyshorteners
Flask
flask-restful

Here's an explanation of each dependency:

appsignal: The AppSignal Python agent that helps us monitor our application's performance.
opentelemetry-instrumentation-redis: The OpenTelemetry instrumentation for Redis to instrument our Redis calls.
opentelemetry-instrumentation-flask: The OpenTelemetry instrumentation for Flask to instrument our Flask application.
redis: The supported Redis client for Python that we will use to interact with Redis.
pyshorteners: A library that helps us shorten URLs.
Flask: The Flask web framework that we will use to build the URL shortener.
flask-restful: A library that helps us build RESTful APIs with Flask.

Note: Keep in mind that the AppSignal Python agent does not work on Windows. If you are using Windows, you can use the Windows Subsystem for Linux (WSL) to run the AppSignal agent.

Now, let’s install the dependencies:

> pip install -r requirements.txt

The AppSignal agent needs an API key to connect to the AppSignal service. You can get your API key by signing up for an AppSignal account — there's a free trial available.

Once you have an account, you can configure the AppSignal agent via this command:

> python -m appsignal install

A series of prompts will ask for your API key and application name. You can use the default values or provide your own. This install command will create an __appsignal__.py file in your project with the configuration for the AppSignal agent.

You can now inspect the __appsignal__.py file that was created in your project.

Alternatively, you can create the __appsignal__.py file manually with the following content:

# __appsignal__.py
from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="url-shortener",
    push_api_key="YOUR_API_KEY",
)

All set! Now, let’s instrument Redis in our application.

Instrumenting Redis

With the AppSignal agent installed and configured, we can now instrument our application.

Create a shorten-url.py file with the following instructions:

# shorten-url.py
import appsignal
from openTelemetry.instrumentation.redis import RedisInstrumentor

# open telemetry instrumentation
appsignal.start()
RedisInstrumentor().instrument()

The appsignal.start() function will start the AppSignal agent and configure it with the settings from the __appsignal__.py file. Be sure to put these lines at the top of your Python file to ensure the AppSignal agent starts before any other code is executed.

Once the AppSignal agent starts, we can use the RedisInstrumentor class from the opentelemetry-instrumentation-redis package to instrument our Redis calls. The instrument() method will configure the Redis client and start collecting metrics.

Note: If you do not see any metrics in the AppSignal dashboard, it is likely because either the AppSignal agent is not running or the Redis instrumentation is not working.

To troubleshoot the instrumentation, you can use the following command:

> python -m appsignal diagnose

This command will check the configuration of the AppSignal agent and the instrumentation of the Redis client. If there are any issues, the command will output an error message that you can use to fix the problem.

Alternatively, you can submit your report to the AppSignal support team for help. The report will be made available via a web link you can share with the support team.

Next, we will build an app that shortens URLs using this instrumentation.

Building the URL Shortener

We will use the pyshorteners library to shorten URLs and the redis library to store the data in Redis. In the shorten-url.py file, simply add the following code:

# shorten-url.py
import appsignal
from openTelemetry.instrumentation.redis import RedisInstrumentor

# open telemetry instrumentation
appsignal.start()
RedisInstrumentor().instrument()

import redis
from flask import Flask, request
from flask_restful import Resource, Api

from urllib.parse import urlparse
import pyshorteners

app = Flask(__name__)
api = Api(app)


class ApiResource(Resource):
    def post(self):
        data = request.get_json()['url']

        def is_valid_url(url):
            result = urlparse(data)
            return all([result.scheme, result.netloc])

        if is_valid_url(data) is False:
            raise Exception('Invalid URL.')

        type_tiny = pyshorteners.Shortener()
        client = redis.StrictRedis(host='localhost', port=6379)

        if client.exists(data):
            return {'url': client.get(data).decode('utf-8')}, 200

        else:
            if 'tinyurl.com' in data:
                raise Exception('URL is already shortened.')

            short_url = type_tiny.tinyurl.short(data)
            client.set(short_url, data)
            return {'ur': (short_url)}


api.add_resource(ApiResource, '/')


if __name__ == '__main__':
    app.run(debug=True)

We do validation checks to ensure the URL is legit and not already shortened. If everything checks out, we shorten the URL and store the data in Redis. If the URL already exists in Redis, we return the shortened URL.

The instrumentation we added earlier will monitor the performance of the Redis calls and provide insights. Every Redis call is monitored and we can see the latencies, errors, and other metrics in our dashboard in AppSignal.

To test this out, you can run the following CURL command:

> curl -i -X POST -H 'Content-Type: application/json' -d '{"url": "https://docs.appsignal.com/python/instrumentations/redis.html"}' http://127.0.0.1:5000
{"url": "https://tinyurl.com/22ynqkyf"}
> curl -i -X POST -H 'Content-Type: application/json' -d '{"url": "https://tinyurl.com/22ynqkyf"}' http://127.0.0.1:5000
{"url": "https://docs.appsignal.com/python/instrumentations/redis.html"}

Feel free to repeat the command with different URLs to see how the URL shortener works. You can also check the Redis database to see the data stored there:

> redis-cli GET https://tinyurl.com/22ynqkyf
"https://docs.appsignal.com/python/instrumentations/redis.html"

That’s it! You have successfully instrumented Redis in a Python application using AppSignal.

You can now monitor the performance of your Redis calls and get insights, which we will cover in the next section.

Check Latencies in AppSignal

If you go to your AppSignal dashboard, you will see the performance metrics for API calls under Performance:

There is an Actions section that shows API call latencies. You can keep track of P95s, P90s, and other metrics to ensure your application is performing well. You can also set up alerts to notify you if the latencies are too high.

To check for Redis latencies, you can go to Slow queries under Performance. There, you will see the latencies of the Redis calls listed by command.

The entire event is captured, including the command, the duration, and the backtrace.

To set up alerts, go to the anomaly detection section of AppSignal:

There, you can configure triggers for latencies in Redis calls. You can set up alerts for P95s and P90s to notify you if the latencies exceed a certain threshold.

Simply click on the Add Trigger button and configure the trigger:

You can set a threshold, duration, and notification method for an alert.

And that's that!

Wrapping Up

In this tutorial, we learned how to instrument Redis in a Python application using AppSignal. We created a simple URL shortener that stores data in a local instance of Redis and monitored the performance of the Redis calls in our dashboard in AppSignal. We also learned how to check Redis call latencies and set up alerts.

Happy coding!

An Introduction to Testing in Python Flask

Roy Tomeij — 2025-04-02T00:00:00+00:00

So, you've built a Flask application — congratulations! You've crafted routes, connected databases, and perhaps even deployed it to a server. But have you tested it thoroughly?

Testing isn't just a checkbox on a developer's to-do list: it's an essential part of building robust and reliable applications.

So, in this article, we'll describe why testing is important for Flask applications and how you can effectively implement tests.

Why Write Tests for Python Flask?

But first of all, why should you test your Flask application?

To ensure code quality

Let's face it: bugs happen no matter how carefully you code, and the more features your application has, the greater the risk that a new change might introduce issues elsewhere. In this context, tests act as a safety net, catching problems before they make it to production. So, by writing tests for your Flask app, you can detect bugs early, ensuring that your code meets the quality standards you and your users expect.

To streamline development

Have you ever added a new feature and accidentally broken something else? Maybe that one "innocent" change caused an entire section of your app to malfunction. If you've experienced that frustration, automated tests are there to back you up. In fact, tests speed up development cycles by identifying issues as soon as they arise, especially when integrated into Continuous Integration (CI) processes.

Also, CI helps run your tests automatically whenever there's a new change in the codebase, ensuring that problems are identified and fixed before they accumulate. This means less time debugging and more time building the features your users will love.

To boost confidence in changes

Making changes to an existing codebase can sometimes feel like walking on eggshells, so you might wonder: "Will this new feature break existing functionality?". Well, a comprehensive test suite alleviates this anxiety because with well-written tests you can make changes confidently, knowing that if any breaking changes occur, they will be detected immediately. So, tests give you the assurance that your code is behaving correctly, even after refactoring or adding new features.

What to Test

Testing is an essential part of the development cycle, but it requires time and effort (often limited resources). Also, while testing everything can be overwhelming, it's usually not practical or necessary. So where should you focus your efforts when testing a web application?

Here are some key components of a Flask application that should be covered by tests:

Unit tests for views and routes

Your views and routes are the entry points to your application because they are where requests from users are handled. Unit tests should ensure that each route returns the correct HTTP response codes, renders the right templates, and provides the expected data. This kind of testing makes sure that individual parts of your application are working correctly in isolation.

Testing business logic

Business logic is the core of your application, as it includes calculations, data processing, and decision-making that empowers your application's functionality.

So, testing your business logic ensures that your helper functions, models, and services perform correctly across a variety of inputs, covering edge cases and typical scenarios. This prevents subtle bugs that may not be immediately visible in the UI.

Testing database interactions

If your application interacts with a database (and let's be honest, most apps do), testing those interactions is fundamental. In particular, you need to ensure that data is being correctly saved, retrieved, updated, and deleted. So, whether you're using Flask-SQLAlchemy or another ORM, tests should cover your data's integrity and verify that CRUD operations function as expected. Database tests can catch issues like incorrect queries, foreign key violations, and data inconsistencies.

Integration and functional tests

Unit tests are great, but they only verify the individual parts of your app in isolation. Integration tests take things a step further by ensuring that all these parts work correctly together. They verify that your routes, databases, and other components interact as expected.

Functional tests go even further, as they simulate user interactions and workflows, such as submitting forms, handling authentication, and verifying permissions. This helps ensure that your application delivers a seamless experience for end-users.

A Popular Python Testing Framework: `pytest`

When it comes to testing in Python, pytest represents a modern and concise framework very popular among developers. It's simple to use, yet powerful enough to handle complex testing needs. With its straightforward syntax and extensive plugin ecosystem, pytest makes writing tests less of a chore and more of an integral part of development.

pytest also provides several useful features, such as fixtures, parameterized tests, and assertion introspection, which make testing more efficient and powerful for developers.

Fixtures

Fixtures allow you to set up a specific state for your tests, like initializing a database or creating necessary test data. This makes your tests cleaner by handling repetitive setup tasks, allowing you to focus on the actual test logic rather than boilerplate code.

For example, if you need to create a test client or seed a database before each test, fixtures can do this automatically.

Parameterized Tests

Parameterized tests let you run the same test function with different inputs, which can significantly reduce code duplication. This helps ensure that your function works correctly across a range of inputs without you needing to write multiple test functions manually.

Assertion introspection

When an assertion fails, pytest provides detailed information about what went wrong. Instead of just telling you that an assertion failed, it shows the values involved, making debugging much easier. This means you spend less time figuring out why a test failed and more time fixing the problem.

Best Practices for Testing in Flask

Let's now provide some guidelines on best practices for testing Flask applications.

Maintain Readable Tests

Your tests are code too, and they deserve the same care and attention as your application code. So, write readable and self-explanatory test cases by using meaningful names for test functions and variables, and structuring your tests logically.

As a rule of thumb, a test case should ideally tell a story: "Given this input, the application should behave like this", so avoid overly complex or tightly coupled tests that are hard to understand and maintain. Readable tests are easier to debug and maintain, and they help other developers (or future you) understand what you're trying to verify.

Keep Tests Isolated

Isolation is key in testing web applications. In particular, each test should run independently of other ones to avoid side effects and ensure accurate results. For example, changes made to a database during one test shouldn't affect another; this means, for example, resetting the database between tests or using mock objects. Isolation, in fact, makes it easier to identify the root cause of any issues and prevents a test from passing or failing due to external factors.

Use Mocking for External Dependencies

Your Flask application may rely on external services like third-party APIs, payment gateways, or microservices. Testing these dependencies directly can lead to flaky tests due to network issues, service downtime, or rate limits. By mocking, you can simulate the behavior of these external services, allowing your tests to be fast, reliable, and independent of external factors.

Focus on Edge Cases and Error Handling

While it's important to test the expected "happy path" scenarios, it's equally important to test how your application handles edge cases and errors. This includes testing with invalid inputs, missing data, or unexpected user behavior. So, thorough testing of edge cases ensures your application is robust, user-friendly, and can gracefully handle exceptions without crashing or exposing sensitive information.

Practical Examples of Testing in Flask

Let's now go through practical examples of how to test a Flask application, by providing different scenarios.

Setting Up Your Flask App for Testing

First, to make testing easier, it's important to structure your repository in a logical way. Here's a common structure for a Flask project:

my_flask_app/
├── app/
│   ├── __init__.py
│   ├── routes.py
│   └── models.py
├── tests/
│   ├── __init__.py
│   ├── test_routes.py
│   └── test_models.py
├── venv/
├── requirements.txt
├── config.py
└── run.py

app/: This directory contains your main application code, including routes, models, and other core functionality.
tests/: This directory contains all your test files. Each major component of your application should have corresponding test files to ensure comprehensive coverage.
venv/: The virtual environment directory. This folder should typically not be committed to version control (.gitignore it).
requirements.txt: A file that lists all dependencies for your project.
run.py: The script used to start your Flask application.

By following this structure, you create a clean separation between your application code and your test code, which helps in maintaining and scaling your project effectively.

Then, you can create a virtual environment to manage dependencies for your Flask project.

On Windows:

> python -m venv venv
> venv\Scripts\activate

On macOS/Linux:

$ python3 -m venv venv
$ source venv/bin/activate

Assuming you've already installed Flask, you can now install pytest:

pip install pytest

Then ensure your Flask app is structured in a way that makes testing feasible. Typically, this means your app should be an importable module. Here's a basic structure:

from flask import Flask

def create_app():
    app = Flask(__name__)

    @app.route('/')
    def index():
        return 'Hello, World!'

    return app

This structure allows you to easily create multiple instances of your Flask app, which is helpful for testing.

We can now go through some basic testing examples.

Writing Unit Tests for Routes

Let's write a test for the / route using pytest:

import pytest
from app import create_app

@pytest.fixture
def client():
    app = create_app()
    app.config['TESTING'] = True

    with app.test_client() as client:
        yield client

def test_index_route(client):
    response = client.get('/')
    assert response.status_code == 200
    assert b'Hello, World!' in response.data

In this test:

We use a pytest fixture to create a test client, which allows us to simulate HTTP requests to our Flask app without starting the server.
We set the app configuration to 'TESTING' = True to indicate that we're running in a test environment, which can help with debugging and error handling.
We send a GET request to the / route.
We assert that the response status code is 200 OK, and that the response data contains the expected text 'Hello, World!'.

Testing Forms and POST Requests

Suppose you have a simple form that submits a user's name:

@app.route('/greet', methods=['POST'])
def greet():
    name = request.form['name']
    return f'Hello, {name}!'

Here's how you can test the form submission:

# test_app.py
def test_greet_route(client):
    response = client.post('/greet', data={'name': 'Alice'})
    assert response.status_code == 200
    assert b'Hello, Alice!' in response.data

In this test:

We send a POST request to the /greet endpoint with form data route ({'name': 'Alice'}).
We assert that the response is successful (status_code == 200).
We verify that the response includes the personalized greeting ('Hello, Alice!').

Writing Parameterized Tests

Suppose we want to test the /greet route with multiple names to ensure it handles different inputs correctly. We can use parameterized tests for this case, like so:

import pytest

@pytest.mark.parametrize("name, expected_greeting", [
    ("Alice", b"Hello, Alice!"),
    ("Bob", b"Hello, Bob!"),
    ("Charlie", b"Hello, Charlie!")
])

def test_greet_route_parametrized(client, name, expected_greeting):
    response = client.post('/greet', data={'name': name})
    assert response.status_code == 200
    assert expected_greeting in response.data

In this parameterized test, we use the @pytest.mark.parametrize decorator to run the same test function with different inputs (name) and expected outputs (expected_greeting).

This helps ensure that the /greet route works correctly for various names without duplicating code.

Testing Database Interactions with Test Isolation

As discussed above, testing database interactions is very important for applications that rely on data storage and retrieval. In this case, you should maintain isolation in testing as it ensures that each test runs in a clean environment, unaffected by the outcomes of other tests.

Let's show how we can provide isolation while testing a database.

Setting Up the Database for Testing

Suppose you have a simple User model in a Flask application using SQLAlchemy:

# models.py
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True, nullable=False)

    def __repr__(self):
        return f'<User {self.username}>'

And in the create_app function, you can initialize the database:

# app/__init__.py
from flask import Flask
from .models import db, User

def create_app():
    app = Flask(__name__)
    app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///:memory:'
    app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False

    db.init_app(app)

    @app.route('/add_user/<username>')
    def add_user(username):
        user = User(username=username)
        db.session.add(user)
        db.session.commit()
        return f'User {username} added.'

    @app.route('/users')
    def list_users():
        users = User.query.all()
        return ', '.join([user.username for user in users])

    return app

In this setup:

We use an in-memory SQLite database (sqlite:///:memory:) for testing purposes.
We have two routes: one to add a user and another to list all users.

Writing Tests with Isolation

To ensure test isolation, you can set up the database afresh for each test. Here's how you can do it using fixtures in pytest:

# tests/test_models.py
import pytest
from app import create_app
from app.models import db, User

@pytest.fixture
def app():
    app = create_app()
    app.config['TESTING'] = True
    app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///:memory:'

    with app.app_context():
        db.create_all()
        yield app
        db.session.remove()
        db.drop_all()

@pytest.fixture
def client(app):
    return app.test_client()

def test_add_user(client):
    response = client.get('/add_user/Alice')
    assert response.status_code == 200
    assert b'User Alice added.' in response.data

    response = client.get('/users')
    assert b'Alice' in response.data

def test_user_list_is_empty(client):
    response = client.get('/users')
    assert response.status_code == 200
    assert response.data == b''

In these tests:

The app Fixture sets up the Flask application and initializes the database for each test. In particular, by using db.create_all() and db.drop_all(), we ensure that each test has a fresh database.
The client Fixture provides a test client for sending HTTP requests to our Flask app.
test_add_user tests that a user can be added and then retrieved from the database.
test_user_list_is_empty tests that the user list is empty when no users have been added.

By dropping and recreating the database between tests, we ensure that tests do not affect each other's outcomes, achieving test isolation.

Final Advice on Database Testing: Cleaning Up After Tests

Always ensure that any resources initialized during a test are properly cleaned up afterward to save resources. In the example above, we:

Used db.session.remove() to remove the database session. This method removes the current SQLAlchemy session associated with an application context, ensuring that any pending transactions are rolled back, and the session is removed from the session registry. Note that if you don't remove the session after a test, the session might retain state (like uncommitted transactions) that could affect subsequent tests.
Dropped all tables after the tests using db.drop_all(). This method removes all schema objects (tables, indexes, constraints), returning the database to an empty state. This guarantees that each test starts with no pre-existing data or schema changes that could influence its behavior.

Some of the consequences of not cleaning up after testing are:

Interdependent tests: Tests might pass or fail depending on the order they are run, leading to unreliable test results.
Resource exhaustion: Open sessions or connections can accumulate, potentially causing the database or application to run out of resources.
Hard-to-debug failures: Side effects from previous tests can cause failures in unrelated tests, making debugging more difficult.

Using Mocking for External Dependencies

Let's consider a common scenario — your application sends emails using an external email service when users register:

# app/routes.py
from flask import Blueprint, request, render_template
import external_email_service

bp = Blueprint('routes', __name__)

@bp.route('/register', methods=['GET', 'POST'])
def register():
    if request.method == 'POST':
        email = request.form['email']
        # Assume user registration logic here
        result = external_email_service.send_welcome_email(email)
        if result:
            return 'Registration successful!', 200
        else:
            return 'Failed to send welcome email.', 500
    return render_template('register.html')

We could test it with mocking like so:

# tests/test_routes.py
import pytest
from unittest.mock import patch
from app import create_app

@pytest.fixture
def client():
    app = create_app()
    app.config['TESTING'] = True

    with app.test_client() as client:
        yield client

@patch('app.routes.external_email_service')
def test_register_route_success(mock_email_service, client):
    mock_email_service.send_welcome_email.return_value = True

    response = client.post('/register', data={'email': 'user@example.com'})
    assert response.status_code == 200
    assert b'Registration successful!' in response.data
    mock_email_service.send_welcome_email.assert_called_once_with('user@example.com')

@patch('app.routes.external_email_service')
def test_register_route_email_failure(mock_email_service, client):
    mock_email_service.send_welcome_email.return_value = False

    response = client.post('/register', data={'email': 'user@example.com'})
    assert response.status_code == 500
    assert b'Failed to send welcome email.' in response.data

In this test:

We use unittest.mock.patch to replace the send_welcome_email function with a mock that simulates the external email service.
We test both scenarios: where the email is sent successfully and when the email service fails (response.status_code == 200 and response.status_code == 500)
We assert that the mock was called with the correct parameters, ensuring our code interacts with the external service as expected.

NOTE: The unittest.mock module is included in the standard library from Python 3.3 and above.

Focusing on Edge Cases and Error Handling

Let's consider another common scenario — your web application provides a login route that should handle invalid credentials gracefully. This could be coded like so:

# app/routes.py
@bp.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        username = request.form['username']
        password = request.form['password']
        # Assume user authentication logic here
        if username == '' or password == '':
            return 'Username and password are required.', 400
        if not authenticate_user(username, password):
            return 'Invalid credentials.', 401
        return 'Login successful!', 200
    return render_template('login.html')

Here's how we could test it:

# tests/test_routes.py
def test_login_missing_credentials(client):
    response = client.post('/login', data={'username': '', 'password': ''})
    assert response.status_code == 400
    assert b'Username and password are required.' in response.data

def test_login_invalid_credentials(client):
    response = client.post('/login', data={'username': 'user', 'password': 'wrongpass'})
    assert response.status_code == 401
    assert b'Invalid credentials.' in response.data

def test_login_success(client):
    # Assuming 'authenticate_user' will return True for these credentials during testing
    with patch('app.routes.authenticate_user', return_value=True):
        response = client.post('/login', data={'username': 'user', 'password': 'pass'})
        assert response.status_code == 200
        assert b'Login successful!' in response.data

In this example, we ensure the application responds appropriately to different user inputs by testing different scenarios:

We test the case where the username and password are missing, expecting a 400 Bad Request response.
We test with incorrect credentials, expecting a 401 Unauthorized response.
We mock the authenticate_user function to return True and test a successful login.

And that's it!

Wrapping Up

Testing isn't just about finding bugs; it's about building confidence in your code and making your development process more efficient and reliable.

By writing tests for your Flask application, you ensure code quality, streamline development, and can make changes without fear of breaking things. By using tools like pytest and best practices such as test isolation and readability, adding tests to your workflow becomes a manageable and rewarding task.

The next time you find yourself hesitating to write tests, remember: the effort you invest in testing today will save you countless hours and headaches in the future, making your Flask application robust, maintainable, and enjoyable to work on.

Happy coding!

An Introduction to Flask-SQLAlchemy in Python

Roy Tomeij — 2025-02-26T00:00:00+00:00

Efficient management of database interactions is one of the most important tasks when building Python web applications. SQLAlchemy — a comprehensive SQL toolkit and Object-Relational Mapping (ORM) library — is widely used in the Python ecosystem for this purpose.

However, integrating SQLAlchemy with Flask can be a complex process. So developers created Flask-SQLAlchemy to provide a seamless way to use SQLAlchemy in Flask applications, making database management a straightforward process.

In this article, we introduce SQLAlchemy and Flask-SQLAlchemy, highlighting their key features. We'll also create a basic Flask app to use Flask-SQLAlchemy for database management.

What is SQLAlchemy?

Let's start by describing what SQLAlchemy is and what its strengths are.

SQLAlchemy is an open source SQL toolkit and ORM library for Python, created to let objects be objects, and tables be tables. It provides developers with tools to work with databases such as Oracle, DB2, MySQL, PostgreSQL, and SQLite at both high and low levels of abstraction.

Its two primary components are the core and ORM:

SQLAlchemy Core: This provides a full suite of well-organized database interaction tools. In particular, it's a low-level SQL toolkit that offers an abstraction layer allowing developers to work directly with relational databases by using Python. It contains the so-called "SQL Expression Language", a toolkit on its own that provides a system for constructing SQL expressions.
SQLAlchemy ORM: The ORM layer builds on top of the Core and provides a higher-level, more abstracted interface for interacting with databases. In practice, it maps Python classes to database tables and instances of these classes to rows in those tables, enabling developers to work with database records as if they were Python objects.

Here's a high-level schema of these two components:

So, basically, if you like writing SQL directly and want fine-grained control over database interactions, then Core is for you. With the ORM, you do not need to write SQL clauses: it allows you to define database models as Python classes and interact with them by using object-oriented programming concepts like inheritance and relationships.

Now let's look at both in more detail.

Using SQLAlchemy ORM

One of the primary features of SQLAlchemy ORM is its ability to map database tables to Python classes seamlessly, allowing developers to manipulate database records using object-oriented paradigms. This approach simplifies database interactions by enabling you to work directly with Python objects instead of writing SQL queries, making code more maintainable and readable.

The ORM also provides a rich set of tools for managing relationships between entities, handling complex queries, and automatically synchronizing the state of in-memory objects with the database. Additionally, SQLAlchemy ORM supports lazy loading and eager loading, offering efficient data retrieval mechanisms tailored to specific use cases. These features, combined with the ability to declaratively define schemas, make the ORM a powerful tool for developers seeking to integrate databases into their Python applications with minimal boilerplate.

Here's how to create a table, insert a record, and query data with the ORM:

from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import declarative_base, sessionmaker

# Define the database engine
engine = create_engine('sqlite:///:memory:', echo=True)

# Define the base class for ORM models
Base = declarative_base()

# Define the ORM model
class User(Base):
    __tablename__ = 'users'

    id = Column(Integer, primary_key=True)
    name = Column(String)
    age = Column(Integer)

# Create the table in the database
Base.metadata.create_all(engine)

# Create a session to interact with the database
Session = sessionmaker(bind=engine)
session = Session()

# Insert a new user
new_user = User(name="John Doe", age=30)
session.add(new_user)
session.commit()

# Query the database
user = session.query(User).filter_by(name="John Doe").first()
print(f"User: {user.name}, Age: {user.age}")

Now onto SQLAlchemy Core.

Using SQLAlchemy Core

SQLAlchemy Core, on the other hand, offers a lower-level approach that provides direct control over database operations, providing features like schema definition, a powerful SQL expression language, and the engine for database connectivity.

The schema/types system in SQLAlchemy Core allows developers to define the structure of their database tables explicitly, including data types, constraints, and indexes, using Python constructs. This ensures a consistent and clear definition of the database schema that can be programmatically manipulated.

The SQL expression language is the central feature of SQLAlchemy Core, enabling developers to build SQL queries dynamically and expressively using Python. This language allows for the construction of complex queries and data manipulations while maintaining full control over the generated SQL, making it ideal for developers who need fine-grained control over their database interactions.

Finally, the engine in SQLAlchemy Core is responsible for establishing and managing connections to the database, executing SQL statements, and handling transactions. It provides a robust and flexible foundation for working with various database backends, offering both synchronous and asynchronous interaction modes.

Here's how to create a table, insert a record, and query data using the Core:

from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, insert, select

# Define the database engine
engine = create_engine('sqlite:///:memory:', echo=True)

# Define the metadata object for schema management
metadata = MetaData()

# Define the table using SQLAlchemy Core
users_table = Table(
    'users', metadata,
    Column('id', Integer, primary_key=True),
    Column('name', String),
    Column('age', Integer)
)

# Create the table in the database
metadata.create_all(engine)

# Insert a new user using Core
with engine.connect() as connection:
    insert_stmt = users_table.insert().values(name="John Doe", age=30)
    connection.execute(insert_stmt)

# Query the database using Core
with engine.connect() as connection:
    select_stmt = select(users_table).where(users_table.c.name == "John Doe")
    result = connection.execute(select_stmt)
    user = result.fetchone()
    print(f"User: {user.name}, Age: {user.age}")

Now, let's define what Flask-SQLAlchemy is and why you should use it.

What is Flask-SQLAlchemy in Python?

Flask-SQLAlchemy is a Flask extension that integrates SQLAlchemy with Flask, simplifying the process of database management in Flask applications. It combines the power and flexibility of SQLAlchemy with the simplicity and ease of use that Flask developers expect.

By providing a layer of abstraction over SQLAlchemy, in fact, Flask-SQLAlchemy makes it easier to set up, configure, and manage databases in Flask-based projects.

Key Features of Flask-SQLAlchemy for Python

Flask-SQLAlchemy comes with several key features that make it an excellent choice for database management in Flask applications, including:

Easy configuration: Flask-SQLAlchemy simplifies the configuration of SQLAlchemy with Flask. You can configure your database connection and other settings through Flask’s configuration system, making it easy to manage different environments (for example: development, testing, and production environments).
Automatic tables creation: If you are connecting to a database that already has tables, Flask-SQLAlchemy can automatically create tables based on your defined models when the application starts. This reduces the need for manual table creation and synchronization.
Integration with Flask’s application context: Flask-SQLAlchemy leverages Flask's app context, ensuring that your database session is properly managed across requests. This simplifies transaction management and ensures that your database operations are handled correctly in a web environment.
Simplified querying: With Flask-SQLAlchemy, querying the database becomes straightforward, as it extends SQLAlchemy’s query interface to work seamlessly within Flask. This allows you to use familiar Flask patterns when interacting with your database.

How Flask-SQLAlchemy Simplifies Database Management

Flask-SQLAlchemy reduces the complexity of setting up and using SQLAlchemy with Flask by:

Streamlining configuration and setup: With Flask-SQLAlchemy, you don’t need to manually initialize and configure SQLAlchemy. The extension takes care of this by automatically binding SQLAlchemy to your Flask application and using Flask’s configuration to manage database settings.
Providing simplified session management: In a web application, managing database sessions can be tricky. Flask-SQLAlchemy automatically manages the database session for each request, ensuring that sessions are properly opened and closed, reducing the risk of transaction errors and memory leaks.
Enhancing development speed: By abstracting away much of the boilerplate code needed to set up and interact with a database, Flask-SQLAlchemy allows developers to focus more on building features rather than on database management. The extension’s tight integration with Flask’s development ecosystem also means that it plays well with other Flask extensions and tools.

Setting Up Flask-SQLAlchemy

So, now that we've presented Flask-SQLAlchemy, let's provide a practical example of how to use it in a Flask application.

You can find all the code stored in this public repo.

Prerequisites

To replicate this example, your system must satisfy the following prerequisites:

Ubuntu (any recent version) for Windows Subsystem for Linux (WSL): The steps in this tutorial are tailored for Ubuntu in WSL, but they are adaptable to other Linux distributions, especially not under WLS, or for different Operating Systems (but you'll need to adapt the code).
Python 3.6+ and pip: Flask-SQLAlchemy requires Python 3.6 or higher. We’ll also install everything we need with pip.
MySQL: The steps in this guide are tested with MySQL installed (on Ubuntu for WSL).
Basic knowledge of Python and Flask: This tutorial requires familiarity with virtual environments, Flask application structure, and basic terminal commands.

Requirements

For Flask to do its job, we first need to create a new database in MySQL.

If you are unfamiliar with it, you can follow this guide.

Step 1: Launch MySQL

Launch MySQL by typing:

$ sudo service mysql start

NOTE: This is particular for Ubuntu on WSL. On Ubuntu (not under WSL) the command changes ($ sudo systemctl mysql start).

Step 2: Access MySQL

Access MySQL by typing:

$ mysql -u root -p

NOTE: Here you'll be asked to insert the password you set when you installed MySQL.

Step 3: Create the Database

Everything is now set up to create a database. You can create it like so:

CREATE DATABASE flask_db;

NOTE: You will have to insert this database name in the Flask app later.

Then, create a user and set its password:

CREATE USER 'flaskuser'@'localhost' IDENTIFIED BY 'your_password';

NOTE: You will have to insert this user and password in the Flask app later.

Finally, grant privileges to the user:

GRANT ALL PRIVILEGES ON flask_db.* TO 'flaskuser'@'localhost';

FLUSH PRIVILEGES;

Installation and Configuration

Now, let's install all the necessary libraries for using Flask and create a web app example using FlaskSQL-Alchemy.

Step 1: Create the Project Directory

First, create the main folder that will host the project via the terminal:

$ mkdir flask_sqlalchemy_example
$ cd flask_sqlalchemy_example

Step 2: Create and Activate a Virtual Environment

Before installing the needed libraries, we suggest creating a virtual environment:

$ python3 -m venv venv
$ source venv/bin/activate

Step 3: Install Flask, Flask-SQLAlchemy, and Other Libraries

You can now install everything that's needed for this tutorial:

$ pip install Flask Flask-SQLAlchemy PyMySQL cryptography Flask-WTF

Step 4: Create the Structure of the Project

Finally, to recreate the demo project, create the following project structure:

flask_sqlalchemy_example/
│
├── app/
│   ├── __init__.py
│   ├── models.py
│   ├── routes.py
│   ├── forms.py
│   ├── templates/
│       └── index.html
│
├── venv/
│   └── ... (virtual environment files)
│
├── config.py
└── run.py

Create the Flask App

Now everything is set up to create the Flask app.

The application we're creating leverages Flask-SQLAlchemy to build a lightweight web application that handles user data management. In particular, we'll create a web app where users can compile a form and input their username and email address.

Once they fill out the form and submit it, the application validates the data. If the input is valid, the app adds a new user record to the MySQL database.

Step 1: Create the `config.py` File

First, create the config.py file to hold the application configuration, including the database URI:

import os

basedir = os.path.abspath(os.path.dirname(__file__))

class Config:
    SECRET_KEY = os.environ.get('SECRET_KEY') or 'a_very_secret_key'
    SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://flaskuser:your_password@localhost/flask_db'
    SQLALCHEMY_TRACK_MODIFICATIONS = False

NOTE: Here you have to use the user, password, and database name set when you created the database in MySQL.

Step 2: Create the `init.py` File

Create the __init__.py file inside the app/ directory to initialize the Flask app and SQLAlchemy:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from config import Config

db = SQLAlchemy()

def create_app():
    app = Flask(__name__)
    app.config.from_object(Config)

    db.init_app(app)

    with app.app_context():
        from . import routes, models
        db.create_all()
        routes.register_routes(app)

    return app

Step 3: Create a Model

Now, let's define a model representing a user in app/models.py:

from app import db

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(64), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)

    def __repr__(self):
        return f'<User {self.username}>'

This is a direct representation of a database table within SQLAlchemy’s ORM, allowing for efficient querying and manipulation of user records.

Step 4: Define a Basic Route

In app/routes.py, create a route to display a list of users:

from flask import render_template, redirect, url_for, flash
from app import db
from app.models import User
from app.forms import UserForm

def register_routes(app):
    @app.route('/', methods=['GET', 'POST'])
    def index():
        form = UserForm()
        if form.validate_on_submit():
            username = form.username.data
            email = form.email.data
            user = User(username=username, email=email)
            db.session.add(user)
            db.session.commit()
            flash('User added successfully!', 'success')
            return redirect(url_for('index'))

        users = User.query.all()
        return render_template('index.html', form=form, users=users)

Step 5: Create a form

Create a form for users in the file app/forms.py:

from flask_wtf import FlaskForm
from wtforms import StringField, SubmitField
from wtforms.validators import DataRequired, Email, Length

class UserForm(FlaskForm):
    username = StringField('Username', validators=[DataRequired(), Length(min=1, max=64)])
    email = StringField('Email', validators=[DataRequired(), Email(), Length(max=120)])
    submit = SubmitField('Add User')

Step 6: Create a Template for the Web Application

Create a template for users to interact with the web app. Add their usernames and emails with Jinja like so:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>User Management</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css">
</head>
<body>
    <div class="container mt-4">
        <h1 class="mb-4">User Management</h1>

        <!-- Flash messages -->
        {% with messages = get_flashed_messages(with_categories=true) %}
            {% if messages %}
                <div class="alert alert-{{ messages[0][0] }} alert-dismissible fade show" role="alert">
                    {{ messages[0][1] }}
                    <button type="button" class="close" data-dismiss="alert" aria-label="Close">
                        <span aria-hidden="true">&times;</span>
                    </button>
                </div>
            {% endif %}
        {% endwith %}

        <!-- User Form -->
        <div class="card mb-4">
            <div class="card-body">
                <form method="POST">
                    {{ form.hidden_tag() }}
                    <div class="form-group">
                        {{ form.username.label(class="form-label") }}
                        {{ form.username(class="form-control") }}
                    </div>
                    <div class="form-group">
                        {{ form.email.label(class="form-label") }}
                        {{ form.email(class="form-control") }}
                    </div>
                    <div class="form-group">
                        {{ form.submit(class="btn btn-primary") }}
                    </div>
                </form>
            </div>
        </div>

        <!-- User List -->
        <h2 class="mb-3">List of Users</h2>
        <table class="table table-striped">
            <thead>
                <tr>
                    <th>ID</th>
                    <th>Username</th>
                    <th>Email</th>
                </tr>
            </thead>
            <tbody>
                {% for user in users %}
                <tr>
                    <td>{{ user.id }}</td>
                    <td>{{ user.username }}</td>
                    <td>{{ user.email }}</td>
                </tr>
                {% endfor %}
            </tbody>
        </table>
    </div>

    <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/@popperjs/core@2.9.3/dist/umd/popper.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
</body>
</html>

Step 7: Run the Application

Finally, create the run.py file at the root of the project to run the application:

from app import create_app

app = create_app()

if __name__ == '__main__':
    app.run(debug=True)

To run the app, type the following via CLI:

$ python3 run.py

And here's the expected result:

Wrapping Up

In this article, we've presented SQLAlchemy as a powerful tool to manage databases using Python.

Specifically, using Flask-SQLAlchemy makes it easier to manage databases as this extension leverages SQLAlchemy specifically for Flask apps.

The example we've run through shows how easy it is to use this library.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Integrate AppSignal with AWS Fargate in Python Flask

Roy Tomeij — 2025-02-12T00:00:00+00:00

In this tutorial, we’ll show you how to integrate AppSignal with a Flask application running on AWS Fargate.

Fargate is a serverless container service that allows you to run Docker containers in the cloud. By integrating AppSignal with AWS Fargate, you can monitor the performance of your Flask application and get insights.

Set Up

The Flask application we'll use in this tutorial is a simple weather API that returns random weather data. We’ll build the cloud infrastructure using the AWS Cloud Development Kit (CDK) and deploy the Flask application to AWS Fargate. The app will run from a Docker image hosted on Amazon Elastic Container Registry (ECR).

Once the CDK stack is deployed, we’ll monitor the Flask application using AppSignal. The Open Telemetry instrumentation will have a custom span we can use to track the performance of the randomly generated weather data.

You can follow along with the code presented here or by cloning the GitHub repository.

PIP packages

The weather API will have the following dependencies:

appsignal: The AppSignal Python agent.
opentelemetry-instrumentation-flask: The OpenTelemetry instrumentation for Flask.
Flask: The Flask web framework.

These requirements can be installed via the requirements.txt file inside the docker image. To keep it simple, we will use the python:3.12-slim image as the base image for the Flask application. The docker container can run on any computer that has Docker installed, and it can be deployed to AWS Fargate.

But first, spin up the CDK project using the Python template. The infrastructure code will be the one to build the docker image inside an app folder.

AWS CDK

Be sure to have the AWS CDK installed on your computer. We will assume you have Python 3.6 or later installed.

Create a new project folder and initialize the CDK project using the Python template:

mkdir flask-fargate-api
cd flask-fargate-api
cdk init app --language python
./venv/Scripts/activate
pip install -r requirements.txt
cdk bootstrap
cdk synth

The activate script will activate the virtual environment for your project. The cdk synth command will verify the CDK project is working correctly.

Open the app.py file in the flask-fargate-api folder and make sure your account details are correct.

FlaskFargateApiStack(
    app,
    "flask-fargate-api-stack",
    env=cdk.Environment(account='1234567890', region='us-east-1')
)

The stack ID flask-fargate-api-stack will be the name of the stack used by CloudFormation. You can change the name to something else if you like. But do not change the stack ID once the stack has been deployed because it will bomb.

Open the flask-fargate-api-stack.py file and add the following code to create the VPC, ECS cluster, and Fargate service:

from aws_cdk import (
    Stack,
    aws_ec2 as ec2,
    aws_ecs as ecs,
    aws_ecs_patterns as ecs_patterns,
    aws_logs as logs
)
from constructs import Construct


class FlaskFargateApiStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        vpc = ec2.Vpc(
            self,
            "vpc"
        )

        cluster = ecs.Cluster(
            self,
            "cluster",
            vpc=vpc
        )

        service = ecs_patterns.ApplicationLoadBalancedFargateService(
            self,
            "api",
            cluster=cluster,
            cpu=256,
            memory_limit_mib=512,
            desired_count=1,
            public_load_balancer=True,

            task_image_options={
                "environment": {
                    "ADDRESS": "0.0.0.0",
                    "PORT": "80"
                },
                "image": ecs.ContainerImage.from_asset(
                    directory="app"
                ),
                "container_port": 80,
                "log_driver": ecs.AwsLogDriver(
                    stream_prefix="api",
                    log_retention=logs.RetentionDays.ONE_DAY
                )
            }
        )

        service.target_group.configure_health_check(
            path="/health"
        )

The task image has environment variables for the address and port. This will allow the loopback address to be used inside the container. The public_load_balancer will give us a public URL to access the Flask application.

The VPC allows the CDK to grant network access to the Fargate service. This allows access to the ECR repository and the ECS cluster. The app itself doesn’t exist yet, but the CDK will build the Docker image and push it to ECR. The health check is used to monitor the health of the Fargate service during deployment.

We are not able to deploy the stack yet, because it needs the Docker image to be built and pushed to ECR. Let's do that next.

Weather API

Create the missing app folder in the root of the project. Inside the app folder, create a new file called app.py and add the following code:

import appsignal
from appsignal import set_category
from opentelemetry import trace

# open telemetry instrumentation
appsignal.start()
tracer = trace.get_tracer(__name__)

from flask import Flask, jsonify
import random
import os

app = Flask(__name__)

weather_data = [
    {"city": "New York", "temperature": 77, "condition": "Cloudy"},
    {"city": "Los Angeles", "temperature": 70, "condition": "Sunny"},
    {"city": "Chicago", "temperature": 73, "condition": "Sunny"},
    {"city": "Houston", "temperature": 81, "condition": "Cloudy"},
    {"city": "Phoenix", "temperature": 92, "condition": "Sunny"},
]


@app.route('/weather', methods=['GET'])
def get_weather():
    # start a custom span
    with tracer.start_as_current_span("random_weather"):
        set_category("get_weather")
        return jsonify(random.choice(weather_data))


@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "ok"})


if __name__ == '__main__':
    port = os.getenv('PORT', 5000)
    host = os.getenv('ADDRESS', 'localhost')

    app.run(debug=True, host=host, port=port)

The tracer object creates a custom span called random_weather. This span will track the performance of the random weather data. Keep in mind this could be an API call, a database query, or any other operation that needs to be monitored. The set_category function sets the category of the span to get_weather.

The appsignal.start() function must be called before the Flask application is imported. This will start the AppSignal agent and configure the OpenTelemetry instrumentation for Flask.

To activate this instrumentation, create an __appsignal.py__ file in the app folder and add the following code:

from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="flask-fargate-api",
    # Please do not commit this key to your source control management system.
    # Move this to your app's security credentials or environment variables.
    # https://docs.appsignal.com/python/configuration/options.html#option-push_api_key
    push_api_key="your-push-api-key"
)

Next, create the requirements.txt file in the app folder and add the following dependencies:

appsignal
opentelemetry-instrumentation-flask
Flask

Create the Dockerfile in the app folder and add the following code:

FROM python:3.12-slim

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r requirements.txt

CMD ["python", "app.py"]

This Dockerfile will create the Docker image for the Flask application to run on AWS Fargate. The command python app.py will run the Flask application when the container is started. You can test the Flask application locally or via the Docker container.

That’s it for the Flask application. The next step is to build the Docker image and push it to ECR via the CDK.

Deploy the CDK Stack

Drum roll, please! Fire up the CDK stack using the following command:

cdk deploy

If there is an issue with the Docker image, be sure to have Docker installed and running on your computer. Docker must also have access to ECR to push the image.

If it’s having trouble signing in to ECR, run the following command:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 1234567890.dkr.ecr.us-east-1.amazonaws.com

Be sure to replace 1234567890 with your AWS account number. This will allow Docker to push the image to ECR.

The deployment should take no more than five minutes. If it is taking longer, kill it in CloudFormation and delete the stack. Troubleshoot the issue and try again. AWS charges for the Fargate service and ECR, so do not leave the stack running if it is not working.

At the end, the CDK will output the public URL for the Flask application. Look for the api URL in the output, which looks like this:

Outputs:
flask-fargate-api-stack.api = http://api-1234567890.us-east-1.elb.amazonaws.com

Now grab the URL and test the Flask application via CURL or a web browser. The /weather endpoint should return random weather data. The /health endpoint should return a status of ok.

Hit the /weather endpoint a few times to generate some data. Next, we will monitor the Flask application using AppSignal.

AppSignal Monitoring of the Python Flask Application

Once you hit the endpoint enough times, go to the AppSignal dashboard under your account. The application should be listed under flask-fargate-api. The name matches the name configuration in the __appsignal__.py file. Click on the application to view the dashboard.

Under Performance, you should see Actions, Slow Events, and Graphs. Click on Actions and then the GET /health endpoint. Then, you can click on 'Slow Events' to see your custom span random_weather.

The random_weather span should show average duration, throughput, and overall performance impact. This is the custom span we created to monitor the performance of the random weather data.

Note the span breakdown, which includes impact, meaning the endpoint spent all its time generating random weather data.

You may also track the performance of each call for the custom span.

The breakdown of the span itself is also available. This shows a timeline of the HTTP request and the time spent generating the random weather data.

And that's that!

Wrapping Up

In this tutorial, we showed you how to integrate AppSignal with a Flask application running on AWS Fargate. We built the cloud infrastructure using the AWS CDK and deployed the application. We then monitored the Flask application using AppSignal and created a custom span to track the performance of the random weather data.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Monitor the Performance of Your Python Flask Application with AppSignal

Roy Tomeij — 2025-01-29T00:00:00+00:00

When a system runs slowly, our first thought might be that it’s failing. This common reaction underscores a key point:

Performance is crucial for an application's readiness and maturity.

In the world of web applications, even milliseconds matter. Performance impacts user satisfaction and operational efficiency, making it a critical factor.

In this article, we'll show you how to use AppSignal to monitor and improve the performance of your Flask applications.

Flask Performance Monitoring Essentials: Key Metrics

Achieving optimal performance in Flask applications involves a comprehensive approach. This means building applications that not only run efficiently but also maintain this efficiency as they scale. Key metrics provide the data needed to guide our optimization efforts. Let’s explore some of these essential metrics:

Response Time: This is a direct indicator of user experience, measuring the time taken for a user's request to be processed and the response to be sent back. In a Flask application, factors such as database queries, view processing, and middleware operations can influence response times.
Throughput: Throughput refers to the number of requests your application can handle within a given timeframe. Higher throughput indicates better performance under load.
Error Rates: The frequency of errors (4xx and 5xx HTTP responses) can indicate issues with code, database queries, or server configuration. Monitoring error rates helps to quickly identify and fix problems that could degrade user experience.
Database Performance Metrics: These include the number of queries per request, query execution time, and the efficiency of database connections. Optimizing these metrics can significantly improve overall application performance.
Handling Concurrent Users: The ability to efficiently serve multiple users accessing your Flask application simultaneously is critical. Monitoring how well your application handles concurrent users helps ensure it can scale effectively without delays.

What We Will Build

In this article, we'll construct a Flask-based blog application ready for high-traffic events, integrating AppSignal to monitor, optimize, and ensure it scales seamlessly under load.

Here's the codebase and the branch appsignal-integration is where we have AppSignal integrated with our Flask app. You can access the live site.

Prerequisites

To follow along, you'll need:

Prepare the Project

Now let's create a directory for our project, clone it from GitHub, and install all the requirements:

mkdir flask-performance && cd flask-performance
python3.12 -m venv venv
source venv/bin/activate
git clone -b main https://github.com/amirtds/myblog
cd myblog
python3.12 -m pip install -r requirements.txt
npm run build:css
python run.py

Now visit 127.0.0.1:8000.

Install AppSignal

Now we are going to install appsignal and opentelemetry-instrumentation-flask in our project.

Before installing these packages, log in to AppSignal using your credentials. After picking the organization, click on Add app on the top right of the navigation bar. Select Python as the language and note the push-api-key.

Make sure your virtual environment is activated and run the following commands in the project:

python3.12 -m pip install appsignal==1.3.6
python3.12 -m appsignal install --push-api-key [YOU-KEY]
python3.12 -m pip install opentelemetry-instrumentation-flask==0.46b0

Provide the app name to the CLI prompt. After finishing the installation, you should see a new file called __appsignal__.py in your project.

Now let's create an .env file in the project root and add APPSIGNAL_PUSH_API_KEY=YOUR-KEY (remember to change the value to your actual key). Next, let's change the content of the __appsignal__.py file to the following:

# __appsignal__.py
import os
from appsignal import Appsignal

# Load environment variables from the .env file
from dotenv import load_dotenv
load_dotenv()

# Get APPSIGNAL_PUSH_API_KEY from environment
push_api_key = os.getenv('APPSIGNAL_PUSH_API_KEY')

appsignal = Appsignal(
    active=True,
    name="myblog",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
)

And update the __init__.py to the following:

from flask import Flask
from __appsignal__ import appsignal # new line
from opentelemetry.instrumentation.flask import FlaskInstrumentor # new line
from opentelemetry import trace # new line

def create_app():
    appsignal.start() # new line
    app = Flask(__name__)
    app.static_folder = 'static'

    # Instrument Flask with OpenTelemetry
    FlaskInstrumentor().instrument_app(app) # new line

    from . import routes
    app.register_blueprint(routes.bp)

    return app

We have imported appsignal and started it using the configuration we used in __appsignal__.py.

Monitor Concurrent Users

Now let's look at monitoring concurrent users.

Scenario: Optimizing for High Load on Blog Posts

As we anticipate increased traffic on our Flask-based blog application, we want to ensure our site remains fast and responsive. We've had traffic spikes that led to slow load times and occasional downtime. We're determined to avoid these issues by thoroughly testing and optimizing our site beforehand. We'll use Locust to simulate user traffic and AppSignal to monitor our application's performance.

Creating a Locust Test for Simulated Traffic

First, we'll create a locustfile.py that simulates users navigating through critical parts of our blog: the homepage and individual post pages. This simulation helps us understand how our site performs under pressure.

Create locustfile.py in the project root:

# locustfile.py
from locust import HttpUser, between, task
import random

class BlogUser(HttpUser):
    wait_time = between(1, 3)  # Users wait 1-3 seconds between tasks
    post_urls = ["/post/post1", "/post/post2", "/post/post3", "/post/post4", "/post/post5", "/post/post6"]

    @task
    def index_page(self):
        self.client.get("/")

    @task(3)
    def view_post(self):
        post_url = random.choice(self.post_urls)
        self.client.get(post_url)

In this locustfile.py, users primarily visit the post pages, occasionally returning to the homepage. This pattern mimics realistic user behavior on a blog.

Before running Locust, ensure you have posts available in your Flask app. You can check this by visiting the homepage and individual post pages.

Defining Acceptable Response Times

Before we start, let's define what we consider acceptable response times. For a smooth user experience, we aim for:

Homepage: under 1 second
Post page: under 1.5 seconds

These targets ensure users experience minimal delay, keeping their engagement high.

Conducting the Test and Monitoring Results

With our Locust test ready, we run it to simulate the 500 users and observe results in real time. Here's how:

Start the Locust test by running locust -f locustfile.py in your terminal, then open http://localhost:8089 to set up and start the simulation. Set the Number of Users to 500 and set the host to http://127.0.0.1:8000
Monitor performance in both Locust's web interface and AppSignal. Locust shows us request rates and response times, while AppSignal provides deeper insights into our Flask app's behavior under load.

After running Locust, you can find information about the load test in its dashboard:

Now go to your application page in AppSignal. Under the Performance section, click on Actions and you should see something similar to the following:

Mean: This is the average response time for all the requests made to a particular endpoint. It provides a general idea of how long it takes for the server to respond. In our context, any mean response time greater than 1 second could be considered a red flag, indicating that our application's performance might not meet user expectations for speed.
90th Percentile: This is the response time at the 90th percentile. For example, for GET /, if we have 7 ms, it means 90% of requests are completed in 7 ms or less.
Throughput: This is the number of requests handled per second.

Now let's click on the Graphs option under Performance. You should see something like the following:

Analyzing and Improving Performance

We need to prepare our site for increased traffic as response times might exceed our targets. Here's a simplified plan:

Database queries: Slow queries often cause performance issues. We will delve into this in the next section.
Static assets: Ensure they're properly cached. Use a CDN for better delivery speeds.
Application resources: Sometimes, the solution is as straightforward as adding more RAM and CPUs.

Track Errors

Let's introduce an error in one of the routes. For example, we can create a route that divides by zero, which will cause a ZeroDivisionError. In the end of the routes.py file, let's add the following:

# Introduce a route that causes an error
@bp.route('/cause-error')
def cause_error():
    # This will cause a ZeroDivisionError
    1 / 0

Now visit http://127.0.0.1:8000/cause-error. We should see a ZeroDivisionError in the dashboard.

Let's head to the dashboard in AppSignal. On the left sidebar, click on Errors -> Issue List. Here you should see all the errors that occur to users when they visit specific URLs. By clicking on each issue, you should see more detailed information.

It is highly recommended that you integrate AppSignal with your GitHub repo. When an error occurs, an issue will be created in the repo automatically so you don't lose track of it.

No additional steps are required beyond the initial setup to begin logging and managing errors.

In the Graphs we can see the Error rate, Count, and Throughput:

Monitor Performance

In the Performance section of your dashboard, click on the Issues list to see a detailed breakdown of all visited URLs along with key metrics:

Mean: This metric indicates the average response time for requests to a specific endpoint. It gives us a sense of the overall speed of our server's responses. For our blog, if the mean response time exceeds 1 second, it could be a warning sign that our performance may not meet user expectations.
Throughput: This metric shows the number of requests being processed per second. High throughput with acceptable response times indicates good performance.
Impact: This reflects the significance of a particular action on the overall application performance, based on its usage relative to other actions.

Additionally, we can visualize our application's performance through various graphs:

This data is crucial as it provides insights into how users experience page load times. High response times signal the need for optimization. For instance, if multiple requests exceed 1 second, we should investigate and improve those areas to enhance performance.

By using these insights from AppSignal, we can continually refine and optimize our Flask application to ensure it delivers a fast and smooth user experience.

Anomaly Detection and Alerts

We can also set up triggers to alert us to high resource usage on our platform. This is crucial for taking action before our site goes down due to a spike in traffic. Additionally, this is important because when a critical function fails, we may see a surge in errors. Anomaly detection can notify us before this impacts more users.

Let's create two triggers: one for high memory usage when it exceeds 70% and another for error rates when they surpass 20%.

Ensure the Email notification option is checked to receive notifications when an anomaly occurs.

In the Issues list, you can view all the anomalies that have occurred based on the conditions you defined.

Uptime Monitoring

Uptime monitoring is essential to ensure your blog remains accessible and performs well. Here's why you might need uptime monitoring:

Post-Deployment Issues: After deploying updates, there might be changes in URLs or resources, such as CSS files, leading to errors and a broken appearance. This not only affects the aesthetics but can also frustrate users.
System Downtimes: Various issues, like database outages or failures in third-party services, can cause your site to go down. Downtimes disrupt user experience and can damage your platform's reputation.

Uptime monitoring not only confirms that your site is operational but also alerts you immediately if it goes down.

Features of Uptime Monitoring

Regional Response Times: AppSignal's uptime check shows response times from different regions, providing insights into global performance. This is particularly useful for ensuring a consistent user experience across various geographical locations.
Public Status Page: Set up a public status page to communicate real-time site status, reducing the number of support tickets as users can verify the site status themselves.

In your Uptime Monitoring dashboard, you can add new URLs by clicking on Add Uptime Monitor. For instance, you can create monitors for your blog's critical resources and main URLs.

By setting up these alerts and monitors, you ensure that your Flask blog application remains robust and performs well, even under high traffic or unexpected issues.

Host Monitoring

In this section, we can observe how our VM resources are being utilized by our Flask application. The metrics section in AppSignal provides visualizations and graphs, making it easier to understand resource usage.

Monitoring Resource Usage

Using AppSignal's Host Monitoring, we can track various aspects of our VM's performance. Here are some of the key metrics we can visualize:

CPU Usage: This metric shows how much CPU is being used by our application. High CPU usage can indicate that our application is under heavy load or that processes need optimization.
Memory Usage: Monitoring memory usage helps us understand how much memory our application consumes. If memory usage is consistently high, optimizing the application or increasing the available memory might be necessary.
Disk I/O: Disk I/O metrics show how much data is being read from or written to the disk. High disk I/O can be a sign of heavy data processing tasks or inefficient data handling.

Importance of Resource Monitoring

Resource monitoring is crucial for maintaining the performance and reliability of our Flask application. By keeping an eye on these metrics, we can:

Identify Bottlenecks: High resource usage can indicate performance bottlenecks that need to be addressed.
Optimize Performance: Understanding resource consumption helps in optimizing the application for better performance.
Prevent Downtime: Monitoring helps in proactively managing resources, preventing potential downtimes due to resource exhaustion.

By using these insights from AppSignal, we can ensure that our Flask blog application remains performant and reliable.

Wrapping Up

Monitoring the performance of your Flask application is crucial for ensuring a smooth and responsive user experience, especially as traffic increases. By integrating AppSignal, we can gain valuable insights into various performance metrics, identify potential bottlenecks, and take proactive measures to optimize our application.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

How to Use Regular Expressions in Python

Roy Tomeij — 2025-01-15T00:00:00+00:00

Regular expressions, commonly known as regex, are a tool for text processing and pattern matching.

In Python, the re module offers a robust implementation of regex, allowing developers to handle complex text manipulation efficiently.

In this article, we'll get to grips with regular expressions and provide practical code examples — from the basic to more advanced — so you can understand how to use regex in Python.

Basics of Regular Expressions

Regular expressions are sequences of characters that define search patterns. They can be used for a variety of tasks, such as searching, editing, and manipulating text.

In this section, let's explore some basic components of regex.

Literals

Literals are the simplest form of regex patterns because they match the exact characters in a search string.

For example, if you want to find the exact word "cat" in the text "The cat sat on the mat", you'll have to use a literal that matches the pattern cat.

Metacharacters

In regular expressions, metacharacters are symbols with special meanings and purposes:

. matches any character, except on a new line.
^ matches the start of a string.
$ matches the end of a string.
* matches 0 or more repetitions of the preceding element.
+ matches 1 or more repetitions of the preceding element.
? matches 0 or 1 repetition of the preceding element.
{} matches a specific number of repetitions of the preceding element.

Character Classes

Character classes match any character inside a given set. Common examples are:

[abc]: matches any of the characters a, b, or c.
\d matches any digit (equivalent to [0-9]).
\w matches any word character (alphanumeric plus underscore).
\s matches any whitespace character.

Quantifiers

Quantifiers specify the number of a character's or group's occurrences:

*: 0 or more.
+: 1 or more.
?: 0 or 1.
{n}: Exactly n.
{n,}: n or more.
{n,m}: Between n and m.

Check out other examples with tutorials.

The `re` Module in Python

To work with regex in Python, we use the module re (provided by the standard Python library, so you don't need to install it separately).

In this section, we'll provide some basic examples of how to use the module re and its functionalities.

The `re.search()` Function

The re.search() function searches for the first match of the regex pattern in a string.

Suppose, for example, that you want to extract any numbers in a string. We could write the following Python code:

import re

# Define the pattern
pattern = r"\d+"
# Define the text to analyz
text = "There are 123 apples."
# Apply the regex
match = re.search(pattern, text)
print(match.group())

This results in:

Let's explain the pattern = r"\d+":

The prefix r defines a raw string that treats backslashes (\) as literal characters and not as escape characters.
\d matches any digit (equivalent to [0-9]).
+ indicates that the previous pattern (the digits) must appear one or more times.

Finally, if the re.search() function finds a match, the match variable returns a match object. So, the group() method of the match object returns the substring corresponding to the pattern found.

The `re.match()` Function

The re.match() function checks for a match only at the beginning of a string.

For example, if we want to search for a numerical match at the beginning of the previously defined text string, we could write the following:

import re

# Define the pattern
pattern = r"\d+"
# Define the text to analyze
text = "There are 123 apples."
# Apply the regex
match = re.match(r"\d+", text)
print(match)

And, as we would expect, here's the result:

None

In this case, there's no match with the applied regex rules in the provided string.

The `re.findall()` Function

The re.findall() function finds all matches of the regex pattern in a string, returning the output in a list.

For example, if you want to find all the numerical patterns in a string, you could write something like:

import re
pattern = r"\d+"
text = "There are 123 apples in 2 trees."
matches = re.findall(pattern, text)
print(matches)

And the result is:

['123', '2']

In this case, two numerical expressions are found in the string.

The `re.sub()` Function

The re.sub() function replaces a matched regex pattern with a replacement string.

For example, if you want to replace "123" with "many":

import re

# Define the pattern
pattern = r"\d+"
# Define the text to analyze
text = "There are 123 apples."
# Apply the regex
replaced_text = re.sub(r"\d+", "many", text)
print(replaced_text)

This results in:

There are many apples.

Note that the re.sub() function applies to all the matches it finds. So, for example, the following code sample:

import re

# Define the pattern
pattern = r"\d+"
# Define the text to analyze
text = "There are 123 apples in 3 trees."
# Apply the regex
replaced_text = re.sub(r"\d+", "many", text)
print(replaced_text)

Results in:

There are many apples in many trees.

When To Use Each Function

re.match(), re.search(). and re.findall() are similar. So when should we use one over the other?

re.search() searches for the first place where the pattern matches a string. Use it when you need to find the first occurrence of a pattern in a string, regardless of where it is.
re.match() searches for a match only at the beginning of a string. Use it to check if a string starts with a certain pattern.
re.findall() finds all matches of a pattern in a string and returns them as a list of strings. Use it when you need to find all pattern occurrences in a string.

Advanced Regex Techniques for Python

In this section, we'll dive into some more advanced regex techniques for Python.

Lookaheads and Lookbehinds

Lookaheads and lookbehinds are types of zero-width assertions that match a position in a string based on what precedes or follows it (without including the preceding or following elements in the match itself).

A lookahead asserts that a certain pattern must follow the current position in the string but does not include this pattern in the matched result.

A lookbehind, on the other hand, asserts that a certain pattern must precede the current position in the string, but does not include this pattern in the matched result.

Let's consider a lookahead example. We want to match sequences of one or more word characters immediately followed by a period:

import re

pattern = r"\w+(?=\.)"
text = "This is a test. Followed by another test."
matches = re.findall(pattern, text)
print(matches)

This results in:

['test', 'test']

Why is this the result? Because the word "test" is followed by a dot, and the sequence is matched two times in the text variable. We use the r"\w+(?=\.)" pattern, because:

\w+ matches one or more word characters (letters, digits, and underscores).
(?=\.) is a positive lookahead asserting that the word characters must be followed by a period. However, the period itself is not included in the match result because lookaheads are zero-width assertions.

Note: A positive lookahead (syntax: (?=...)) asserts that what follows the current position matches the specified pattern, while a negative lookahead (syntax: (?!...)) states that what follows the current position does not match the specified pattern.

Now, let's consider a lookbehind example. We want to match one or more word characters immediately preceded by a whitespace character. So, consider the following:

import re

pattern = r"(?<=\s)\w+"
text = "This is a test. Followed by another test."
matches = re.findall(pattern, text)
print(matches)

This results in:

['is', 'a', 'test', 'Followed', 'by', 'another', 'test']

Let's explain the code:

(?<=\s) is a positive lookbehind asserting that the current position in the string must be preceded by a whitespace character (\s), such as a space, tab, or new line. The whitespace itself is not included in the match result because lookbehinds are zero-width assertions.
\w+ matches one or more word characters (letters, digits, and underscores).

So, the regex pattern (?<=\s)\w+ looks for sequences of one or more word characters that are immediately preceded by a whitespace character. And since we used the re.findall() function, the code prints all the words of the text except for "This" because it is at the beginning of the sentence (so it has no whitespace before it).

Non-capturing Groups

Non-capturing groups (syntax: (?:...)) are used to group parts of a pattern without capturing them for later reference. This technique is useful when you need to group parts of your pattern to apply quantifiers, alternation, or other regex operations but do not need to store the match for later use.

Let's consider a practical example. Suppose you have a sequence of numbers like "123-45-6789". You want to intercept the part of the number that has two digits. Here's how to do so with regex in Python:

import re

pattern = r"(?:\d{3})-(\d{2})-(\d{4})"
text = "123-45-6789"
match = re.search(pattern, text)
print(match.group(1))

This results in:

Let's explain the code:

(?:\d{3}) is a non-capturing group that matches exactly three digits.
- matches the hyphen character literally.
(\d{2}) is a capturing group that matches exactly two digits and captures this match for later reference.
(\d{4}) is another capturing group that matches exactly four digits and captures this match for later reference.
match.group(1) retrieves the content of the first capturing group from the match object. So, in this case, it prints the two digits captured by the first capturing group (\d{2}), which is 45.

Real-World Examples and Use Cases

Now, let's examine some real-world scenarios to give a more practical idea of how and when to use regex in Python.

Data Validation

Regular expressions are commonly used to validate data formats such as email addresses, phone numbers, and dates.

For example:

pattern = r"^[\w\.-]+@[\w\.-]+\.\w+$"
email = "example@example.com"
match = re.match(pattern, email)
print(bool(match))

In this case, the result of the print() function is True, meaning that the string example@example.com has been recognized as an email address by the regex pattern. Here's how this works:

^ asserts the position at the start of the string.
[\w\.-]+ matches one or more word characters (alphanumeric characters and underscores), dots, or hyphens.
@ matches the @ character.
\. matches a literal dot.
\w+ matches one or more word characters (alphanumeric characters and underscores).
$ asserts the position at the end of the string.

So this pattern searches for word characters before and after the @ character. It also searches for a literal dot and other words after it. That's how email addresses are created.

Extracting IP Addresses

Another typical use case is extracting an IP address from a log file.

In Python, we can create it like so:

import re

pattern = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"
log = "User logged in from IP 192.168.0.1"
ip = re.search(pattern, log)
print(ip.group())

Let's explain the pattern:

\b asserts a word boundary, ensuring the IP address is not part of a larger string of digits.
\d{1,3} matches 1 to 3 digits.
\. matches a literal dot.

Repeated three times, this pattern is the schema of an IP address.

Performance Considerations

Regular expressions can be computationally expensive, especially with complex patterns.

So, in this section, we'll provide some tips for optimizing regex performance.

Tip 1: Avoid Recompiling Patterns

If you use the same regex multiple times, compile it once with re.compile(). Here's a typical use case:

import re
import time

# Sample texts
texts = ["123", "abc 456", "def 789 ghi"] * 10000

# Pattern without compiling
pattern1 = r"\d+"

start_time = time.time()
for text in texts:
    match = re.search(pattern1, text)
end_time = time.time()
time_without_compile = end_time - start_time

# Compiling the pattern
pattern2 = re.compile(r"\d+")

start_time = time.time()
for text in texts:
    match = pattern2.search(text)
end_time = time.time()
time_with_compile = end_time - start_time

print(f"without compile:{time_without_compile: .3}, with compile:{time_with_compile: .3}")

This results in:

without compile: 0.0286, with compile: 0.0178

In this example, every time the code iterates through the list, the pattern is recompiled if you don't use the re.compile() function, lowering execution time.

Tip 2: Use Specific Patterns

Using specific regex patterns rather than broad ones can significantly improve performance. Specific patterns reduce the number of possible matches the regex engine has to consider, speeding up the matching process.

Let's consider the following:

import re
import time

# Sample texts
texts = ["abc123xyz", "123", "a123b", "x123y", "nonumber", "123", "test"] * 50000

# Less efficient pattern
pattern1 = r".*123.*"

start_time = time.time()
for text in texts:
    match = re.search(pattern1, text)
end_time = time.time()
time_with_less_efficient_pattern = end_time - start_time

# More efficient pattern
pattern2 = r"\b123\b"

start_time = time.time()
for text in texts:
    match = re.search(pattern2, text)
end_time = time.time()
time_with_more_efficient_pattern = end_time - start_time

print(f"time with not efficient pattern: {time_with_less_efficient_pattern: .3}, time with efficient pattern: {time_with_more_efficient_pattern: .3}")

This results in:

time with not efficient pattern:  0.412, time with efficient pattern:  0.385

In this case, we have:

.*123.*: A less efficient pattern because .* matches any character (except a newline) 0 or more times, both before and after "123". The regex engine has to consider a large number of potential matches.
\b123\b is more efficient because it is more specific. \b asserts a word boundary, ensuring that "123" is matched only as a whole word. This reduces the number of possible matches the regex engine needs to evaluate.

Tip 3: Always Use Raw Strings

Using raw strings (by prefixing the pattern with r) is a good practice, especially when dealing with backslashes and escape sequences. When you use raw strings, Python treats backslashes as literal characters rather than escape characters. This can prevent unexpected behavior or errors in your regex patterns and also results in performance improvements.

This example shows that performance is better with raw strings:

import re
import time

# Define a regex pattern without using a raw string
pattern_without_raw = "\d+"

# Define the same regex pattern using a raw string
pattern_with_raw = r"\d+"

# Create some sample text to search
text = "123 456 789 012" * 3000000

# Search using the pattern without raw string
start_time = time.time()
matches_without_raw = re.findall(pattern_without_raw, text)
end_time = time.time()
time_without_raw = end_time - start_time

# Search using the pattern with raw string
start_time = time.time()
matches_with_raw = re.findall(pattern_with_raw, text)
end_time = time.time()
time_with_raw = end_time - start_time

# Print the results and performance comparison
print(f"Time taken without raw string:{time_without_raw: .3}")
print(f"Time taken with raw string: {time_with_raw: .3}")

Here's the result:

Time taken without raw string: 1.65
Time taken with raw string:  1.6

On the side of error management, here's an example:

import re

# Define a regex pattern to match email addresses
pattern_without_raw = "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"

# Sample text containing email addresses
text = "Contact us at email@example.com or visit our website www.example.com"

# Try to find email addresses using the pattern without a raw string
matches_without_raw = re.findall(pattern_without_raw, text)

# Print the matches
print(f"Matches without raw string: { matches_without_raw}")

This results in:

Matches without raw string: []

We haven't used a raw string, so Python treats the backslashes in the pattern as escape characters. This leads to unintended behavior: specifically, \b is interpreted as a backspace character instead of a word boundary, and the pattern fails to match email addresses correctly.

The correct pattern to avoid this unintended behavior is:

pattern = r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"

And that's it for our whistle-stop tour of regex in Python!

Wrapping Up

In this article, we covered the basics, advanced techniques, real-world use cases, and performance optimization strategies of regular expressions in Python.

By understanding and utilizing these concepts, you can handle complex text-processing tasks with ease and precision.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Find and Fix N+1 Queries in Django Using AppSignal

Roy Tomeij — 2024-12-04T00:00:00+00:00

In this article, you'll learn about N+1 queries, how to detect them with AppSignal, and how to fix them to speed up your Django apps significantly.

We'll start with the theoretical aspects and then move on to practical examples. The practical examples will mirror scenarios you might encounter in a production environment.

Let's get started!

What Are N+1 Queries?

The N+1 query problem is a prevalent performance issue in web applications that interact with a database. These queries can cause significant bottlenecks, which intensify as your database grows.

The problem occurs when you retrieve a collection of objects and then access the related objects for each item in the collection. For instance, fetching a list of books requires a single query (1 query), but accessing the author for each book triggers an additional query for every item (N queries).

N+1 problems can also occur when creating or updating data in a database. For example, iterating through a loop to create or update objects individually, rather than using methods like bulk_create() or bulk_update(), can result in excessive queries.

N+1 queries are highly inefficient because executing numerous small queries is significantly slower and more resource-intensive than consolidating operations into fewer, larger queries.

Django's default QuerySet behavior can inadvertently lead to N+1 issues, especially if you're unaware of how QuerySets work. Querysets in Django are lazy, meaning no database queries are executed until the QuerySet is evaluated.

Prerequisites

Ensure you have:

Python 3.9+ and Git installed on your local machine
An AppSignal-supported operating system
An AppSignal account

Note: The source code for this project can be found in the appsignal-django-n-plus-one GitHub repository.

Project Setup

We'll work with a book management web app. The web app is built to demonstrate the N+1 query problem and how to resolve it.

Start by cloning the base branch of the GitHub repo:

$ git clone git@github.com:duplxey/appsignal-django-n-plus-one.git \
    --single-branch --branch base && cd appsignal-django-n-plus-one

Next, create and activate a virtual environment:

$ python3 -m venv venv && source venv/bin/activate

Install the requirements:

(venv)$ pip install -r requirements.txt

Migrate and populate the database:

(venv)$ python manage.py migrate
(venv)$ python manage.py populate_db

Lastly, start the development server:

(venv)$ python manage.py runserver

Open your favorite web browser and navigate to http://localhost:8000/books. The web app should return a JSON list of 500 books from the database.

The Django admin site is accessible at http://localhost:8000/admin. The admin credentials are:

user: username
pass: password

Install AppSignal for Django

To install AppSignal on your Django project, follow the official docs:

Ensure everything works by restarting the development server:

(venv)$ python manage.py runserver

Your app should automatically send a demo error to AppSignal. From this point forward, all your errors will be sent to AppSignal. Additionally, AppSignal will monitor your app's performance and detect any issues.

Web App Logic

The prerequisite to fixing N+1 queries is understanding your app's database schema. Pay close attention to your models' relationships: they can help you pinpoint potential N+1 problems.

Models

The web app has two models — Author and Book — which share a one-to-many (1:M) relationship. This means each book is associated with a single author, while an author can be linked to multiple books.

Both models have a to_dict() method for serializing model instances to JSON. On top of that, the Book model uses deep serialization (serializing the book as well as the book's author).

The models are defined in books/models.py:

# books/models.py
class Author(models.Model):
    first_name = models.CharField(max_length=64)
    last_name = models.CharField(max_length=64)
    birth_date = models.DateField()

    def full_name(self):
        return f"{self.first_name} {self.last_name}"

    def to_dict(self):
        return {
            "id": self.id,
            "first_name": self.first_name,
            "last_name": self.last_name,
            "birth_date": self.birth_date,
        }

    def __str__(self):
        return f"{self.first_name} {self.last_name}"


class Book(models.Model):
    title = models.CharField(max_length=128)
    author = models.ForeignKey(
        to=Author,
        related_name="books",
        on_delete=models.CASCADE,
    )
    summary = models.TextField(max_length=512, blank=True, null=True)
    isbn = models.CharField(max_length=13, unique=True, help_text="ISBN-13")
    published_at = models.DateField()

    def to_dict(self):
        return {
            "id": self.id,
            "title": self.title,
            "author": self.author.to_dict(),
            "summary": self.summary,
            "isbn": self.isbn,
            "published_at": self.published_at,
        }

    def __str__(self):
        return f"{self.author}: {self.title}"

They are then registered for the Django admin site in books/admin.py, like so:

# books/admin.py
class BookInline(admin.TabularInline):
    model = Book
    extra = 0


class AuthorAdmin(admin.ModelAdmin):
    list_display = ["full_name", "birth_date"]
    inlines = [BookInline]


class BookAdmin(admin.ModelAdmin):
    list_display = ["title", "author", "published_at"]


admin.site.register(Author, AuthorAdmin)
admin.site.register(Book, BookAdmin)

Notice that AuthorAdmin uses BookInline to display the author's books within the author's admin page.

Views

The web app provides the following endpoints:

/books/ returns the list of books
/books/<book_id>/ returns a specific book
/books/by-authors/ returns a list of books grouped by authors
/books/authors/ returns the list of authors
/books/authors/<author_id>/ returns a specific author

The links above are clickable if you have the development web server running.

And they're defined in books/views.py like so:

# books/views.py
def book_list_view(request):
    books = Book.objects.all()
    return JsonResponse(
        {
            "count": books.count(),
            "results": [book.to_dict() for book in books],
        }
    )


def book_details_view(request, book_id):
    try:
        book = Book.objects.get(id=book_id)
        return JsonResponse(book.to_dict())
    except Book.DoesNotExist:
        return JsonResponse({"error": "Book not found"}, status=404)


def book_by_author_list_view(request):
    try:
        authors = Author.objects.all()
        return JsonResponse(
            {
                "count": authors.count(),
                "results": [
                    {
                        "author": author.to_dict(),
                        "books": [book.to_dict() for book in author.books.all()],
                    }
                    for author in authors
                ],
            }
        )
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)


def author_list_view(request):
    authors = Author.objects.all()
    return JsonResponse(
        {
            "count": authors.count(),
            "results": [author.to_dict() for author in authors],
        }
    )


def author_details_view(request, author_id):
    try:
        author = Author.objects.get(id=author_id)
        return JsonResponse(author.to_dict())
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)

Great, you now know how the web app works!

In the next section, we'll benchmark our app to detect N+1 queries with AppSignal and then modify the code to eliminate them.

Detect N+1 Queries in Your Django App with AppSignal

Detecting performance issues with AppSignal is easy. All you have to do is use/test the app as you normally would (for example, perform end-user testing by visiting all the endpoints and validating the responses).

When an endpoint is hit, AppSignal will create a performance report for it and group all related visits together. Each visit will be recorded as a sample in the endpoint's report.

Detect N+1 Queries in Views

Firstly, visit all your app's endpoints to generate the performance reports:

Next, let's use the AppSignal dashboard to analyze slow endpoints.

Example 1: One-To-One Relationship (`select_related()`)

Navigate to your AppSignal app and select Performance > Issue list on the sidebar. Then click Mean to sort the issues by descending mean response time.

Click on the slowest endpoint (books/) to view its details.

Looking at the latest sample, we can see that this endpoint returns a response in 1090 milliseconds. The group breakdown shows that SQLite takes 651 milliseconds while Django takes 439.

This indicates a problem because an endpoint as simple as this shouldn't take as long.

To get more details on what happened, select Samples in the sidebar and then the latest sample.

Scroll down to the Event Timeline to see what SQL queries got executed.

Hovering over the query.sql text displays the actual SQL query.

More than 1000 queries were executed:

SELECT * FROM "books_book";
SELECT * FROM "books_author" WHERE "books_author"."id" = 3; -- 3= 1st book's author id
SELECT * FROM "books_author" WHERE "books_author"."id" = 6; -- 6= 2nd book's author id
...
SELECT * FROM "books_author" WHERE "books_author"."id" = n; -- n= n-th book's author id

These are a clear sign of N+1 queries. The first query fetched a book (1), and each subsequent query fetched the book's author's details (N).

To fix it, navigate to books/views.py and modify book_list_view() like so:

# books/views.py
def book_list_view(request):
    books = Book.objects.all().select_related("author")  # modified
    return JsonResponse(
        {
            "count": books.count(),
            "results": [book.to_dict() for book in books],
        }
    )

By utilizing Django's select_related() method, we select the additional related object data (i.e., author) in the initial query. The ORM will now leverage a SQL join, and the final query will look something like this:

SELECT * FROM "books_book"
    INNER JOIN "books_author" ON ("books_book"."author_id" = "books_author"."id")

Wait for the development server to restart and retest the affected endpoint.

After benchmarking again, the response time goes from 1090 to 45, and the number of queries lowers from 1024 to 2. This is a 24x and 512x improvement, respectively.

Example 2: Many-To-One Relationship (`prefetch_related()`)

Next, let's look at the second slowest endpoint (books/by-authors/).

Use the dashboard as we did in the previous step to inspect the endpoint's SQL queries. You'll notice a similar but less severe N+1 pattern with this endpoint.

This endpoint's performance is less severe because Django is smart enough to cache the frequently executed SQL queries, i.e., repeatedly fetching the author of a book. Check out the official docs to learn more about Django caching.

Let's utilize prefetch_related() in books/views.py to speed up the endpoint:

# books/views.py
def book_by_author_list_view(request):
    try:
        authors = Author.objects.all().prefetch_related("books")  # modified
        return JsonResponse(
            {
                "count": authors.count(),
                "results": [
                    {
                        "author": author.to_dict(),
                        "books": [book.to_dict() for book in author.books.all()],
                    }
                    for author in authors
                ],
            }
        )
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)

In the previous section, we used the select_related() method to handle a one-to-one relationship (each book has a single author). However, in this case, we're handling a one-to-many relationship (an author can have multiple books), so we must use prefetch_related().

The difference between these two methods is that select_related() works on the SQL level, while prefetch_related() optimizes on the Python level. The latter method can also be used for many-to-many relationships.

For more information, check out Django's official docs on prefetch_related().

After benchmarking, the response time goes from 90 to 44 milliseconds, and the number of queries lowers from 32 to 4.

Detect N+1 Queries in Django Admin

Discovering N+1 queries in the Django admin site works similarly.

First, log in to your admin site and generate performance reports (for example, create a few authors or books, update, and delete them).

Next, navigate to your AppSignal app dashboard, this time filtering the issues by admin:

In my case, the two slowest endpoints are:

/admin/login
/admin/books/author/<object_id>

We can't do much about /admin/login, since it's entirely handled by Django, so let's focus on the second slowest endpoint. Inspecting it will reveal an N+1 query problem. The author is fetched separately for each book.

To fix this, override get_queryset() in BookInline to fetch author details in the initial query:

# books/admin.py
class BookInline(admin.TabularInline):
    model = Book
    extra = 0

    def get_queryset(self, request):
        queryset = super().get_queryset(request)
        return queryset.select_related("author")

Benchmark once again and verify that the number of queries has decreased.

Wrapping Up

In this post, we've discussed detecting and fixing N+1 queries in Django using AppSignal.

Leveraging what you've learned here can help you significantly speed up your Django web apps.

The two most essential methods to keep in mind are select_related() and prefetch_related(). The first is used for one-to-one relationships, and the second is for one-to-many and many-to-many relationships.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Advanced Open edX Monitoring with AppSignal for Python

Roy Tomeij — 2024-10-30T00:00:00+00:00

In the first part of this series, we explored how AppSignal can significantly enhance the robustness of Open edX platforms. We saw the challenges that Open edX faces as it scales and how AppSignal's features — including real-time performance monitoring and automated error tracking — provide essential tools for DevOps teams. Our walkthrough covered the initial setup and integration of AppSignal with Open edX, highlighting the immediate benefits of this powerful observability framework.

In this second post, we'll dive deeper into the advanced monitoring capabilities that AppSignal offers. This includes streaming logs from Open edX to AppSignal, monitoring background workers with Celery, and tracking Redis queries. We will demonstrate how these features can be leveraged to address specific operational challenges, ensuring that our learning platform remains fail-safe under varying circumstances.

By the end of this article, you will know how to utilize AppSignal to its full potential in maintaining and improving the performance and reliability of your Open edX platform.

Streaming Logs to AppSignal

One of AppSignal's strongest features is centralized log management.

Commonly at Open edX, the support team reports an issue with the site, and an engineer can SSH into the server right away to check for Nginx, Mongo, MySQL, and Open edX Application logs.

A centralized storage place that houses logs without the need for you to SSH into the server is a really powerful feature. We can also set up notifications based on an issue's severity.

Now let's see how we can stream our logs from Open edX to AppSignal.

Create a Source

Under the Logging section, click on Manage sources and create a new source, with HTTP as the platform and JSON as the format. After creating the source, AppSignal provides an endpoint and API KEY that we can POST our logs to.

To have more control over log transmission, we can write a simple Python script that reads logs from our local Open edX, pre-processes them, and moves the important ones to AppSignal. For example, I wrote the following script to move only ERROR logs to AppSignal (skipping INFO and WARNING logs):

import requests
import json
from datetime import datetime
import logging

# Setup logging configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# File to keep track of the last processed line
log_pointer_file = '/root/.local/share/tutor/data/lms/logs/processed.log'
log_file = '/root/.local/share/tutor/data/lms/logs/all.log'

# APpSignal API KEY
api_key = "MY-API-KEY"  # Replace with your actual API key
# URL to post the logs
url = f'https://appsignal-endpoint.net/logs?api_key={api_key}'

def read_last_processed():
    try:
        with open(log_pointer_file, 'r') as file:
            content = file.read().strip()
            last_processed = int(content) if content else 0
            logging.info(f"Last processed line number read: {last_processed}")
            return last_processed
    except (FileNotFoundError, ValueError) as e:
        logging.error(f"Could not read from log pointer file: {e}")
        return 0

def update_last_processed(line_number):
    try:
        with open(log_pointer_file, 'w') as file:
            file.write(str(line_number))
            logging.info(f"Updated last processed to line number: {line_number}")
    except Exception as e:
        logging.error(f"Could not update log pointer file: {e}")

def parse_log_line(line):
    if 'ERROR' in line:
        parts = line.split('ERROR', 1)
        timestamp = parts[0].strip()
        message_parts = parts[1].strip().split(' - ', 1)
        message = message_parts[1] if len(message_parts) > 1 else ''
        attributes_part = message_parts[0].strip('[]').split('] [')
        # Flatten attributes into a dictionary with string keys and values
        attributes = {}
        for attr in attributes_part:
            key_value = attr.split(None, 1)
            if len(key_value) == 2:
                key, value = key_value
                key = key.rstrip(']:').replace(' ', '_').replace('.', '_')  # Replace spaces and dots in keys
                if len(key) <= 50:
                    attributes[key] = value
        # Format the timestamp
        formatted_timestamp = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S,%f').isoformat()[:-3] + 'Z'
        # Add the message and attributes to the log structure
        json_data = {
            "timestamp": formatted_timestamp,
            "group": "openedx",
            "severity": "error",
            "hostname": "tutor",
            "message": message,
        }
        json_data.update(attributes)  # Add the attributes directly to the json_data dictionary
        return json_data

def post_logs(json_data):
    headers = {'Content-Type': 'application/json'}
    response = requests.post(url, json=json_data, headers=headers)
    logging.info(f"Posted log to server; HTTP status code: {response.status_code}, Response: {response.content}")
    return response.status_code

def process_logs():
    last_processed = read_last_processed()
    with open(log_file, 'r') as file:
        for i, line in enumerate(file, 1):
            if i > last_processed:
                json_data = parse_log_line(line)
                if json_data:
                    response_code = post_logs(json_data)
                    if response_code == 200:
                        update_last_processed(i)
                    else:
                        logging.warning(f"Failed to post log, HTTP status code: {response_code}")

if __name__ == '__main__':
    logging.info("Starting log processing script.")
    process_logs()
    logging.info("Finished log processing.")

Here's how the script works:

Log File Management: Tutor saves all of the logs in the /root/.local/share/tutor/data/lms/logs/all.log file. This file contains MySQL, LMS, CMS, Caddy, Celery, and other services. The script uses a pointer /root/.local/share/tutor/data/lms/logs/processed.log file that tracks the last processed line. This ensures that each log is processed only once.
Error Filtering: As mentioned, we only send ERROR logs to AppSignal.
Data Parsing and Formatting: Each error log is parsed to extract key pieces of information, such as the timestamp and error message. The script formats this data into a JSON structure suitable for transmission.
Log Transmission: The formatted log data is sent to AppSignal using an HTTP POST request.

Important: Please make sure you don't send any personally identifiable information to the endpoint.

Now run this script and it should move ERROR logs to AppSignal:

You can also create a new trigger to notify you as soon as a specific event like ERROR happens:

Monitor Celery and Redis using AppSignal

Celery (a distributed task queue) is a vital component of Open edX, responsible for managing background tasks such as grading, certificate generation, and bulk email dispatch. Redis often acts as the broker for Celery, managing task queues. Both systems are essential for asynchronous processing and can become bottlenecks during periods of high usage. Monitoring these services with AppSignal provides valuable insights into task execution and queue health, helping you preemptively address potential issues. Let's see how we can monitor Celery and Redis.

First, install the necessary packages. Add the following to the OPENEDX_EXTRA_PIP_REQUIREMENTS variable in the .local/share/tutor/config.yml file:

- opentelemetry-instrumentation-celery==0.45b0
- opentelemetry-instrumentation-redis==0.45b0

It should look like the following:

OPENEDX_EXTRA_PIP_REQUIREMENTS:
  - appsignal==1.3.0
  - opentelemetry-instrumentation-django==0.45b0
  - opentelemetry-instrumentation-celery==0.45b0
  - opentelemetry-instrumentation-redis==0.45b0

As you can see, we are installing opentelemetry packages for Celery and Redis.

Now, we can instrument Celery with worker_process_init to report its metrics to AppSignal.

Heading back to our dashboard in AppSignal, we should see Celery and Redis reports in the Performance section, with background as the namespace.

For Redis queries, you can click on Slow queries:

Practical Monitoring: Enhancing Open edX with AppSignal

In this section, we'll revisit the initial issues outlined in part one of this series and apply practical AppSignal monitoring solutions to ensure our Open edX platform stays robust and reliable. Here’s a breakdown.

Site Performance Improvement

Let's begin by assessing overall site performance. In the Performance section, under the Issue list, we can see key metrics for all visited URLs:

Response Time: Directly reflects user experience by measuring the time taken to process and respond to requests. Factors influencing this include database queries and middleware operations.
Throughput: Indicates the number of requests handled within a given timeframe.
Mean Response Time: Provides an average response time across all requests to a specific endpoint. Any mean response time over 1 second is a potential concern and highlights areas that need optimization.
90th Percentile Response Time: For example, a 90th percentile response time of 7 ms for GET store/ suggests that 90% of requests complete in 7 ms or less.

Now let's order all the actions based on the mean. Any item higher than 1 second should be considered a red flag:

As we see, Celery tasks to rescore and reset student attempts, LMS requests to show course content, and some APIs are taking more than 1 second. Also, we should note that this is only for one active user. If we have more concurrent users, this response time will go up. Our first solution is to add more resources to the server (CPU and memory) and do another performance test.

After identifying actions with mean response times exceeding 1 second, consider performance optimization strategies such as:

Minimizing JavaScript execution
Using CDNs for static content
Implementing caching techniques.

Server Resource Monitoring

We talked about anomaly detection and host monitoring in the previous article. Let's add triggers for the following items:

CPU usage
Disk usage
Memory usage
Network traffic
Error rate

Custom Metrics

Two really important metrics for our platform are our number of active users and enrollments. Let's see how we can measure these metrics using AppSignal.

First, add increment_counter to common/djangoapps/student/views/management.py and openedx/core/djangoapps/user_authn/views/login.py to track and increment the number of logins and enrollments when there is a new event.

Now let's log in to Open edX and enroll in a course. Next, let's head to our dashboard in AppSignal. Click on Add dashboard, then Create dashboard, and give it a name and description.

Click on Add graph, enter Active Users as the title, select Add Metric and use login_count:

Your dashboard should look like the following:

You can follow the same steps to add a graph for enrollments using an enrollment_count metric.

Ensuring Consistent Styling

To make sure our site's styling stays consistent, let's add a new uptime check for static/tailwind/css/lms-main-v1.css and get notified when a URL is broken:

Email Delivery and Error Handling

In the Error section of the dashboard, we can view all errors, set up notifications for them, and work on fixes as soon as possible to prevent users from being negatively impacted.

Background Job Efficiency for Grading

In the Monitor Celery and Redis section of this article, we saw how to instrument Celery and Redis using AppSignal. Let's follow the same steps to enable AppSignal so we can see graded tasks. In the lms/djangoapps/grades/tasks.py file, add the following lines:

We should now see a couple of items to grade under Performance -> Issue list.

As you can see, recalculate_subsection_grade_v3 (our main grading Celery task) takes 212 milliseconds. For regrading, lms.djangoapps.instructor_task.tasks.reset_problem_attempts and lms.djangoapps.instructor_task.tasks.rescore_problem take 1.77 seconds.

Wrapping Up

In this two-part series, we integrated AppSignal with Open edX to fortify its monitoring capabilities. We started with the basics — setting up and understanding the fundamental offerings of AppSignal, including error tracking and performance monitoring.

In this article, we tackled how to efficiently stream logs from various Open edX services to AppSignal, ensuring all relevant information was centralized and readily accessible. We also monitored crucial asynchronous tasks handled by Celery and Redis.

Finally, we addressed some real-world challenges, such as slow site responses, resource bottlenecks during high enrollment periods, and unexpected issues like broken styling.

By now, you should have a comprehensive understanding of how to leverage AppSignal to not just monitor, but also significantly improve, the performance and reliability of your Open edX platform.

If you have any questions about Open edX or need further assistance, feel free to visit cubite.io or reach out to me directly at amir@cubite.io.

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

How to Use Lambda Functions in Python

Roy Tomeij — 2024-10-16T00:00:00+00:00

Lambda functions in Python are a powerful way to create small, anonymous functions on the fly. These functions are typically used for short, simple operations where the overhead of a full function definition would be unnecessary.

While traditional functions are defined using the def keyword, Lambda functions are defined using the lambda keyword and are directly integrated into lines of code. In particular, they are often used as arguments for built-in functions. They enable developers to write clean and readable code by eliminating the need for temporary function definitions.

In this article, we'll cover what Lambda functions do and their syntax. We'll also provide some examples and best practices for using them, and discuss their pros and cons.

Prerequisites

Lambda functions have been a part of Python since version 2.0, so you'll need:

Minimum Python version: 2.0.
Recommended Python version: 3.10 or later.

In this tutorial, we'll see how to use Lambda functions with the library Pandas: a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation library. If you don't have it installed, run the following:

pip install pandas

Syntax and Basics of Lambda Functions for Python

First, let's define the syntax developers must use to create Lambda functions.

A Lambda function is defined using the lambda keyword, followed by one or more arguments and an expression:

lambda arguments: expression

Let's imagine we want to create a Lambda function that adds up two numbers:

add = lambda x, y: x + y

Run the following:

result = add(3, 5)
print(result)

This results in:

We've created an anonymous function that takes two arguments, x and y. Unlike traditional functions, Lambda functions don't have a name: that's why we say they are "anonymous."

Also, we don't use the return statement, as we do in regular Python functions. So we can use the Lambda function at will: it can be printed (as we did in this case), stored in a variable, etc.

Now let's see some common use cases for Lambda functions.

Common Use Cases for Lambda Functions

Lambda functions are particularly used in situations where we need a temporarily simple function. In particular, they are commonly used as arguments for higher-order functions.

Let's see some practical examples.

Using Lambda Functions with the `map()` Function

map() is a built-in function that applies a given function to each item of an iterable and returns a map object with the results.

For example, let's say we want to calculate the square roots of each number in a list. We could use a Lambda function like so:

# Define the list of numbers
numbers = [1, 2, 3, 4]

# Calculate square values and print results
squared = list(map(lambda x: x ** 2, numbers))
print(squared)

This results in:

[1, 4, 9, 16]

We now have a list containing the square roots of the initial numbers.

As we can see, this greatly simplifies processes to use functions on the fly that don't need to be reused later.

Using Lambda Functions with the `filter()` Function

Now, suppose we have a list of numbers and want to filter even numbers.

We can use a Lambda function as follows:

# Create a list of numbers
numbers = [1, 2, 3, 4]

# Filter for even numbers and print results
even = list(filter(lambda x: x % 2 == 0, numbers))
print(even)

This results in:

[2,4]

Using Lambda Functions with the `sorted()` Function

The sorted() function in Python returns a new sorted list from the elements of any iterable. Using Lambda functions, we can apply specific filtering criteria to these lists.

For example, suppose we have a list of points in two dimensions: (x,y). We want to create a list that orders the y values incrementally.

We can do it like so:

# Creates a list of points
points = [(1, 2), (3, 1), (5, -1)]

# Sort the points and print
points_sorted = sorted(points, key=lambda point: point[1])
print(points_sorted)

And we get:

[(5, -1), (3, 1), (1, 2)]

Using Lambda Functions in List Comprehensions

Given their conciseness, Lambda functions can be embedded in list comprehensions for on-the-fly computations.

Suppose we have a list of numbers. We want to:

Iterate over the whole list
Calculate and print double the initial values.

Here's how we can do that:

# Create a list of numbers
numbers = [1, 2, 3, 4]

# Calculate and print the double of each one
squared = [(lambda x: x ** 2)(x) for x in numbers]
print(squared)

And we obtain:

[1, 4, 9, 16]

Advantages of Using Lambda Functions

Given the examples we've explored, let's run through some advantages of using Lambda functions:

Conciseness and readability where the logic is simple: Lambda functions allow for concise code, reducing the need for standard function definitions. This improves readability in cases where function logic is simple.
Enhanced functional programming capabilities: Lambda functions align well with functional programming principles, enabling functional constructs in Python code. In particular, they facilitate the use of higher-order functions and the application of functions as first-class objects.
When and why to prefer Lambda functions: Lambda functions are particularly advantageous when defining short, "throwaway" functions that don't need to be reused elsewhere in code. So they are ideal for inline use, such as arguments to higher-order functions.

Limitations and Drawbacks

Let's briefly discuss some limitations and drawbacks of Lambda functions in Python:

Readability challenges in complex expressions: While Lambda functions are concise, they can become difficult to read and understand when used for complex expressions. This can lead to code that is harder to maintain and debug.
Limitations in error handling and debugging: As Lambda functions can only contain a single expression, they can't include statements, like the try-except block for error handling. This limitation makes them unsuitable for complex operations that require these features.
Restricted functionality: Since Lambda functions can only contain a single expression, they are less versatile than standard functions. This by-design restriction limits their use to simple operations and transformations.

Best Practices for Using Lambda Functions

Now that we've considered some pros and cons, let's define some best practices for using Lambda functions effectively:

Keep them simple: To maintain readability and simplicity, Lambda functions should be kept short and limited to straightforward operations. Functions with complex logic should be refactored into standard functions.
Avoid overuse: While Lambda functions are convenient for numerous situations, overusing them can lead to code that is difficult to read and maintain. Use them judiciously and opt for standard functions when clarity is fundamental.
Combine Lambda functions with other Python features: As we've seen, Lambda functions can be effectively combined with other Python features, such as list comprehensions and higher-order functions. This can result in more expressive and concise code when used appropriately.

Advanced Techniques with Lambda Functions

In certain cases, more advanced Lambda function techniques can be of help.

Let's see some examples.

Nested Lambda Functions

Lambda functions can be nested for complex operations.

This technique is useful in scenarios where you need to have multiple small transformations in a sequence.

For example, suppose you want to create a function that calculates the square root of a number and then adds 1. Here's how you can use Lambda functions to do so:

# Create a nested lambda function
nested_lambda = lambda x: (lambda y: y ** 2)(x) + 1

# Print the result for the value 3
print(nested_lambda(3))

You get:

Integration with Python Libraries for Advanced Functionality

Many Python libraries leverage Lambda functions to simplify complex data processing tasks.

For example, Lambda functions can be used with Pandas and NumPy to simplify data manipulation and transformation.

Suppose we have a data frame with two columns. We want to create another column that is the sum of the other two. In this case, we can use Lambda functions as follows:

# Create the columns' data
data = {'A': [1, 2, 3], 'B': [4, 5, 6]}

# Create data frame
df = pd.DataFrame(data)

# Create row C as A+B and print the dataframe
df['C'] = df.apply(lambda row: row['A'] + row['B'], axis=1)
print(df)

And we get:

That's it for our whistle-stop tour of Lambda functions in Python!

Wrapping Up

In this article, we've seen how to use Lambda functions in Python, explored their pros and cons, some best practices, and touched on a couple of advanced use cases.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Integrating Open edX with AppSignal

Roy Tomeij — 2024-10-02T00:00:00+00:00

Imagine stepping into the role of a DevOps engineer at an online learning company that utilizes Open edX as its core Learning Management System (LMS). As the platform scales to accommodate more learners, a myriad of challenges begin to surface:

Frequent reports of sluggish platform responses
Issues with password reset emails
Discrepancies in grading reflected on progress pages

These are just the tip of the iceberg. It's pivotal that you provide timely reports on site performance and error tracking in real time, and fix any issues before they affect a significant user base.

In this article (part one of a two-part series), we will see how AppSignal can be harnessed to enhance the robustness of Open edX. Leveraging AppSignal's comprehensive suite of features, we aim to not only monitor — but also significantly improve — platform performance.

What is Open edX?

Open edX is at the forefront of the e-learning landscape, powering Learning Management Systems for millions worldwide. With a high demand for online education, maintaining a high-quality learning experience becomes increasingly crucial.

Why AppSignal for Open edX?

In such a dynamic environment, robust monitoring and error tracking are indispensable. This is where AppSignal shines, offering a powerful toolset for early issue detection, thereby mitigating potential impacts on your learners. By integrating AppSignal with Open edX, you unlock essential features for real-time performance monitoring, automated error tracking, and actionable insights that drive informed decisions for your platform’s evolution.

Throughout this article, we’ll explore practical strategies to leverage AppSignal to build a resilient, fail-safe learning platform that consistently delivers outstanding educational experiences.

Open edX Challenges

Over the last decade of working with Open edX, I've frequently heard the following:

"The site is slow."
"We expect a high spike in enrollments and need more server resources."
"Users apply for password resets but never receive the emails."
"Grading delays are affecting the student progress page."
"The site styling breaks intermittently."
"Students encounter 500 errors without any clear explanation."

These issues underscore the critical need for a robust monitoring platform. Without it, functions like course enrollments could malfunction, overwhelming the support team or deteriorating user experiences. This may potentially cause learners to abandon courses that educators have heavily invested in developing.

The Need for Dedicated Monitoring

Open edX lacks a default system for real-time error tracking, performance monitoring, and user behavior analysis. This gap necessitates a third-party solution that can integrate seamlessly and enhance platform reliability.

Concurrent Users and Performance Issues: Handling large numbers of concurrent users during busy periods is vital for ensuring a smooth, uninterrupted user experience.
Uptime Monitoring: Monitoring downtimes and uptimes is crucial to maintaining a consistent learning environment, and minimizing student frustration due to technical issues.
Anomaly Detection and Error Tracking: Effective monitoring includes identifying unusual behaviors or patterns in user interactions, and providing detailed insights that help to improve educational content and methodologies.

While tools like Sentry and DataDog offer valuable features, they may not fully meet the unique demands of an e-learning environment. This brings us to the distinct advantages of AppSignal.

You can visit Compare AppSignal to Datadog and Compare AppSignal to New Relic to learn about AppSignal's difference to these tools and why it is a better decision to pick AppSignal for Open edX.

How does AppSignal Benefit Open edX?

AppSignal stands out by catering specifically to the needs of platforms like Open edX, providing a robust set of tools designed to tackle the unique challenges of online learning environments:

All-Inclusive Features: AppSignal offers a comprehensive suite of monitoring tools, including error tracking, performance monitoring, and anomaly detection. This holistic approach eliminates the need to manage multiple services.
Real-Time Analytics and Dashboards: With customizable dashboards, AppSignal provides immediate insights into critical metrics such as error rates, uptime, and response times. You can also create custom metrics like enrollment count and visualize these metrics via a custom dashboard.
Ease of Integration: The AppSignal SDK makes integration straightforward, minimizing setup time and allowing for quick implementation of comprehensive monitoring capabilities.

Incorporating AppSignal not only addresses the immediate operational challenges but also enhances the strategic development of Open edX platforms by providing deep, actionable insights into system performance and user engagement.

AppSignal uses the OpenTelemetry protocol, which provides a robust foundation for observability. This integration brings a wealth of out-of-the-box tools and features, enhancing the monitoring capabilities of your Open edX platform.

Prerequisites

To integrate AppSignal with Open edX, we need the following requirements:

Python 3.8 or higher
Django 1.10 or higher
Docker v24.0.5+ (with BuildKit 0.11+)
Docker Compose v2.0.0+
Open edX Koa version or higher
An AppSignal account — sign up for a free 30-day trial
Basic Django and Open edX knowledge

In this article, we will use Tutor to install and customize Open edX.

How to Install Open edX

In this section, we'll quickly review how to install the Open edX Dev instance on the Ubuntu 22.04 operating system.

To Install Tutor on other operating systems, please visit the official Tutor docs.

Open your terminal and run the following commands (before continuing, make sure you have the necessary libraries and applications mentioned in the Prerequisites section):

sudo apt update -y
sudo apt upgrade -y
sudo apt install python3 python3-pip libyaml-dev
pip install "tutor[full]"
tutor local launch

Provide all the necessary information and a correct URL pointing to your server's IP address.

After finishing the deployment you should be able to visit the URL you provided to see your Open edX instance.

Integrate AppSignal

Let's integrate Open edX with AppSignal to start receiving data in our AppSignal Dashboard.

First, go to the AppSignal login page, create a new Python application, and grab your API KEY. We are going to need that in the next step.

Now let's go to where you installed Tutor. Open the file .local/share/tutor/config.yml and add the following libraries to the OPENEDX_EXTRA_PIP_REQUIREMENTS configuration to install all the necessary libraries for AppSignal:

OPENEDX_EXTRA_PIP_REQUIREMENTS:
  - appsignal==1.3.0
  - opentelemetry-instrumentation-django==0.45b0

Next, run tutor config save and relaunch the platform by running tutor local launch. This should install all the necessary packages for you.

After the deployment completes, we'll initialize AppSignal in our Open edX code base and recreate the Open edX Docker images.

Clone the Quince branch of the edx-platform codebase: git clone https://github.com/openedx/edx-platform/tree/open-release/quince.master. Open edx-platform in your favorite editor and follow these instructions:

In the lms directory, create a new file called __appsignal__.py and add the following code:

import os
from openedx.core.djangoapps.site_configuration.models import SiteConfiguration

# Get APPSIGNAL_PUSH_API_KEY from environment
site = SiteConfiguration.objects.get(site__domain="thelearningalgorithm.ai")
APPSIGNAL_PUSH_API_KEY = os.environ.get("APPSIGNAL_PUSH_API_KEY")
os.environ.setdefault("PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION", "python")

from appsignal import Appsignal
appsignal = Appsignal(
    active=True,
    name="openedx",
    push_api_key=APPSIGNAL_PUSH_API_KEY,
)

In your tutor server, create a new environment variable called APPSIGNAL_PUSH_API_KEY and set the AppSignal API key there.

Next, let's start AppSignal in wsgi. Go to lms/wsgi.py and add the following code:

modulestore() # exiting line

# import appsignal
from lms.__appsignal__ import appsignal

# Start Appsignal
appsignal.start()

You can look at this commit to see how I did it.

Learn more about integrating AppSignal with Django.

Now, create a new repo for your edx-platform on GitHub and push your changes there as a new branch. When you are done, run the following command to build a new image for the Open edX container based on your changes:

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY=https://github.com/[YOUR ORGANIZTION]/edx-platform.git --build-arg EDX_PLATFORM_VERSION=[YOUR BRANCH NAME] && tutor local stop && tutor local start -d

If you want to build based on my branch, you can run the following command:

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY=https://github.com/amirtds/edx-platform.git --build-arg EDX_PLATFORM_VERSION=feat/appsignal-integration && tutor local stop && tutor local start -d

That's it! Now go to your Open edX site and do some basic interactions like logging in and viewing course content. You should see that your AppSignal Dashboard starts populating with data from your instance.

In this section, we added AppSignal for the LMS part of Open edX. You can follow the same steps in the CMS directory to add monitoring for the studio.

Track Errors with AppSignal

Now head to the AppSignal Dashboard and click on Errors -> Issue List on the left sidebar. Here, you should see all the errors that occur when users visit specific URLs. By clicking on each issue, you should see more detailed information.

Integrate AppSignal with your GitHub repo so that an issue is created in the repo automatically when an error occurs. This way, you won't lose track of errors.

In the Graphs, we can see the error rate, count, and throughput:

Monitor Performance

In the Performance section, click on the Issues list. Here, you should see a breakdown of all the visited URLs and important metrics, including:

Mean: The average response time for all the requests made to a particular endpoint. It provides a general idea of how long it takes for the server to respond. In our context, any mean response time greater than 1 second could be considered a red flag, indicating that our application's performance might not meet user expectations for speed.
Throughput: This is the number of requests that are being handled per second.
Impact: An action's impact on our application, based on its usage compared to other actions.

We can also see graphs showing our application's performance:

This information is highly important for us since it shows the user experience when loading each page. When numbers are high, we should look for ways to improve the user experience and reduce the mean response time. For example, in our case, we have multiple requests that take longer than 1 second. That means there is room to improve our application's performance.

In the Performance section, we can also monitor background tasks and queries like Celery tasks. We'll look at this in more depth in part two of this series.

Anomaly Detection and Alerts

Here, we can define triggers when there is unusual resource usage on our platform. It is highly important to monitor resource usage and take action so our site doesn't go down when there's a spike in users. Additionally, an important function on the platform might fail, resulting in a high number of errors. By using anomaly detection, we get a notification before it impacts more users.

Let's create 2 triggers: one for high memory usage when it's above 70% and one for % of error rates when they are higher than 20%.

In the Anomaly Detection section on the left sidebar, click on Triggers and follow these steps:

Check the Email notification to receive an email notification when an anomaly happens.

In the Issues list, you can see all anomalies based on defined conditions:

Uptime Monitoring

In my experience with Open edX over the past few years, two scenarios have consistently highlighted the need for robust uptime monitoring:

Post-Deployment Styling Issues: After deployments, CSS links can sometimes change, leading to a 404 error on the CSS resource URL and a broken appearance due to missing styles. This not only impacts the aesthetic appeal of Open edX, but can also confuse and frustrate users.
System Downtimes: Various factors can bring an entire site down, such as database outages or failures in the third-party systems that we've integrated. These downtimes disrupt the learning process and can damage our platform's reputation.

Uptime monitoring is invaluable as it not only confirms our site's operational status, but also alerts us the moment it goes down, allowing us to take quick action to restore functionality.

Regional Response Times: An interesting feature of AppSignal's uptime check is its ability to display response times from different regions. This data provides critical insights into how a site's performance varies globally, which is particularly useful for international educational platforms looking to deliver a consistent user experience across diverse geographical locations.

Public Status Page: You can set up a public status page to communicate real-time site status. This page serves as a reliable source for students to verify whether a site is operational, and can reduce the number of support tickets you get. For instance, you can view an example here.

Simply add a new URL by clicking on Add Uptime Monitor. In this case, I created one for the CSS resource and one for the main LMS URL.

Host Monitoring

Here, we see how our VM resources are being used by our application through visuals and graphs.

Wrapping Up

In this article, we've explored how AppSignal can be integrated with Open edX to bring robust monitoring capabilities to your learning platform. Without requiring extensive configuration beyond an initial setup, AppSignal provides a rich array of features right out of the box. These features enable real-time insights into application performance, error tracking, and uptime monitoring — all essential tools for any DevOps team dedicated to creating a fail-safe learning environment.

In part two of this two-part series, we will dive deeper into the capabilities of AppSignal by demonstrating how to stream logs from Open edX to AppSignal, monitor background tasks with Celery, track Redis queries, and more.

Until then, happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Integrating Stripe Into A One-Product Django Python Shop

Roy Tomeij — 2024-09-04T00:00:00+00:00

In the first part of this series, we created a Django online shop with htmx.

In this second part, we'll handle orders using Stripe.

What We'll Do

We'll integrate Stripe to handle payments securely. This is what we want to achieve:

In the purchase view, we start by creating a Stripe checkout session and redirect the customer to the corresponding URL. This is where we tell Stripe about the product we are selling, its quantity, and where the customer should be redirected to after a successful purchase (the success_url).
The customer fills in their payment details on the Stripe checkout page and completes the payment. Stripe then makes a POST request to a webhook endpoint on our website, where we listen to events and process them accordingly. If the payment is successful, we save the order in our database and notify the customer (and our staff users) about the purchase.
Finally, if the webhook returns a response with a 200 OK HTTP status code, Stripe redirects to the success_url created in the first step.

Setting Up Stripe for Our Django Python Store

We first need to jump over to Stripe and do the following:

1: Create a Stripe Account

Start by creating a Stripe account. For now, you don’t really need to activate your account. You can just work in test mode, which will prevent you from making real payments while testing. Go to the API keys page and retrieve the publishable and secret keys. Save them in your project environment variables (STRIPE_PUBLISHABLE_KEY and STRIPE_SECRET_KEY). We will use these keys to authenticate your Stripe requests.

2: Create Your Product

Create a new product on the products page. Fill out the details and set the payment type to one-off. Your product should look something like this:

Once you press Add product, you should be able to see your product on the product list. If you click on it and scroll down to the Pricing section, you can find the API ID for the price item you created — it should be something like price_3ODP5…. Save it in an environment variable (STRIPE_PRICE_ID): you will need this when creating the Stripe checkout session.

3: Create the Webhook

We need to create a webhook endpoint for Stripe to call when a payment completes. In the webhooks page, choose to test in the local environment. This will allow you to forward the request to a local URL, like http://127.0.0.1:8000. Start by downloading the Stripe CLI. Then, you can:

Log into Stripe

stripe login

Forward events to the webhook endpoint that you will create:

stripe listen --forward-to http://127.0.0.1:8000/webhook
> Ready! Your webhook signing secret is whsec_06531a7ba22363ac038f284ac547906b89e5c939f8d55dfd03a3619f9adc590a (^C to quit)

This ensures that once a purchase is made, Stripe forwards the webhook calls to your local endpoint. The command will log a webhook signing secret, which you should also save as a project environment variable (STRIPE_WEBHOOK_SECRET). This will prove useful for verifying that a request does indeed come from Stripe and that you are handling the right webhook.

By the end of this section, you should have four Stripe environment variables. You can now load them in ecommerce_site/settings.py:

# ecommerce_site/settings.py

import os
from dotenv import load_dotenv
load_dotenv()

STRIPE_PUBLISHABLE_KEY = os.environ.get("STRIPE_PUBLISHABLE_KEY")
STRIPE_SECRET_KEY = os.environ.get("STRIPE_SECRET_KEY")
STRIPE_PRICE_ID = os.environ.get("STRIPE_PRICE_ID")
STRIPE_WEBHOOK_SECRET = os.environ.get("STRIPE_WEBHOOK_SECRET")

Note: We are using python-dotenv to load the environment variables.

Extend the Views

We now need to extend the views to integrate Stripe by creating a checkout session, a successful purchase view, and a webhook view.

1: Create a Stripe Checkout Session

In the purchase view, we'll create a Stripe checkout session if the purchase form is valid:

# ecommerce/views.py
from django_htmx import HttpResponseClientRedirect
from django.conf import settings
import stripe

@require_POST
def purchase(request):
    form = OrderForm(request.POST)
    if form.is_valid():
        quantity = form.cleaned_data["quantity"]

        # replace time.sleep(2) with the following code ⬇️

        # 1 - set stripe api key
        stripe.api_key = settings.STRIPE_SECRET_KEY

        # 2 - create success url
        success_url = (
            request.build_absolute_uri(
                reverse("purchase_success")
            )
            + "?session_id={CHECKOUT_SESSION_ID}"
        )

        # 3 - create cancel url
        cancel_url = request.build_absolute_uri(reverse("home"))

        # 4 - create checkout session
        checkout_session = stripe.checkout.Session.create(
            line_items=[
                {
                    "price": settings.STRIPE_PRICE_ID,
                    "quantity": quantity,
                }
            ],
            mode="payment",
            success_url=success_url,
            cancel_url=cancel_url
        )

        # 5 - redirect to checkout session url
        return HttpResponseClientRedirect(checkout_session.url)
    return render(request, "product.html", {"form": form})

Let’s break this down:

We first set the Stripe API key.
We then create a successful purchase URL pointing to the purchase_success view (which we'll create in the next step). Stripe should automatically populate the CHECKOUT_SESSION_ID.
We create a URL for when a purchase is canceled — for example, when the customer changes their mind. In this case, it’s just the home view.
We create a Stripe checkout session with our price ID (the product identifier) and the quantity the customer wants to purchase.
Stripe returns a session object from which we can extract the URL and redirect the customer. Since this request is coming from htmx, we can’t really use the standard Django redirect function. Instead, we use the django-htmx package, which provides this HttpResponseClientRedirect class.

2: Create the Successful Purchase View

After completing the purchase, Stripe will redirect the customer to our specified success_url. Here, we can handle the post-purchase logic:

from django.shortcuts import redirect

def purchase_success(request):
    session_id = request.GET.get("session_id")
    if session_id is None:
          return redirect("home")

    stripe.api_key = settings.STRIPE_SECRET_KEY
    try:
        stripe.checkout.Session.retrieve(session_id)
    except stripe.error.InvalidRequestError:
        messages.error(request, "There was a problem while buying your product. Please try again.")
        return redirect("home")
    return render(request, "purchase_success.html")

In this view, we first check if the session_id query parameter is present. If it is, we retrieve the corresponding session from Stripe using the secret key and the session_id. We then render the successful purchase template, which looks like this:

# ecommerce/templates/purchase_success.html {% extends "base.html" %} {% block
content %}
<section>
  <header>
    <h2>Thank you for your purchase</h2>
    <p>
      Your purchase was successful. You will receive an email with the details
      of your purchase soon.
    </p>
  </header>
</section>
{% endblock %}

You should also add it to the urlpatterns:

# ecommerce_site/urls.py

# ... same imports as before

urlpatterns = [
    # ... same urls as before
    path("purchase_success", views.purchase_success, name="purchase_success"),  # ⬅️ new
]

3: Create the Webhook View

While the customer is in the purchase process, and before they are redirected to the success view, Stripe will call our webhook endpoint (remember to have the webhook listener running, as explained in the earlier 'Create the Webhook' section of this post):

from django.views.decorators.csrf import csrf_exempt
from django.http import HttpResponse

@csrf_exempt
def webhook(request):
    stripe.api_key = settings.STRIPE_SECRET_KEY
    sig_header = request.headers.get('stripe-signature')
    payload = request.body
    event = None
    try:
            event = stripe.Webhook.construct_event(
                payload, sig_header, settings.STRIPE_WEBHOOK_SECRET
            )
    except stripe.error.SignatureVerificationError:
        # Invalid signature
        return HttpResponse(status=400)

    # Handle the checkout.session.completed event
    if event.type == "checkout.session.completed":
        # TODO: create line orders
        return HttpResponse(status=200)
    return HttpResponse(status=400)

Let’s break this down:

We try to construct a Stripe event from the payload, the signature header, and the webhook secret: the first is used to build the actual event, and the last two variables are relevant to validate the authenticity of the request.
If the signature verification fails, we return a 400 HTTP response. Remember that Stripe is actually calling this endpoint, not our customer, so Stripe will know what to do in this scenario.
We check if the event type is checkout.session.completed, i.e., if a customer successfully paid for our product. For now, we don’t do much else here, but we will process the order in the next step.

Note: A Stripe event can have multiple types but we will only handle completed sessions in this post. However, you can (and should) extend a webhook by following the docs.

You should also add this view to urlpatterns:

# ecommerce_site/urls.py

# ... same imports as before

urlpatterns = [
    # ... same urls as before
    path("webhook", views.webhook, name="webhook"),  # ⬅️ new
]

If everything works well, once you click “buy”, you should be redirected to a Stripe payment page. Since we are in test mode, we can fill in the payment details with dummy data, like a 4242 4242 4242 4242 card:

Once you press Pay, Stripe should call the webhook view and redirect you to the purchase_success view. Congratulations, you have successfully processed a payment with Stripe!

Create the Orders and Notify Users

Once a purchase is completed, we need to do a few things in the webhook view:

Save the order information in our database.
Notify staff users about the recent purchase.
Send a confirmation email to the customer.

Let’s create a LineOrder database model in ecommerce/models.py to store some of the order information:

# ecommerce/models.py

from django.db import models

class LineOrder(models.Model):
    quantity = models.IntegerField()
    name = models.CharField(max_length=255, null=True, blank=True)
    email = models.EmailField(null=True, blank=True)
    shipping_details = models.TextField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"Order {self.id} - {self.quantity} units"

Remember to create and run the migrations:

python manage.py makemigrations # ⬅️ creates the migration files
python manage.py migrate # ⬅️ applies the migrations in the database

We can now create a function to process the orders and call it from the webhook view:

# ecommerce/views.py

@csrf_exempt
def webhook(request):
    # ...same code as before
        if event.type == "checkout.session.completed":
            create_line_orders(event.data.object) # ⬅️ new
            return HttpResponse(status=200)
        return HttpResponse(status=400)

# new ⬇️
def create_line_orders(session: stripe.checkout.Session):
    line_items = stripe.checkout.Session.list_line_items(session.id)
    for line_item in line_items.data:
        LineOrder.objects.create(
            name=session.customer_details.name,
            email=session.customer_details.email,
            shipping_details=session.shipping_details,
            quantity=line_item.quantity,
        )
    mail.send_mail(
        "Your order has been placed",
        f"""
        Hi {session.customer_details.name},
        Your order has been placed. Thank you for shopping with us!
        You will receive an email with tracking information shortly.

        Best,
        The one product e-commerce Team
        """,
        "from@example.com",
        [session.customer_details.email],
    )

    staff_users = User.objects.filter(is_staff=True)
    mail.send_mail(
        "You have a new order!",
        """
            Hi team!
            You have a new order in your shop! go to the admin page to see it.

            Best,
            The one product e-commerce Team
            """,
        "from@example.com",
        [user.email for user in staff_users],
    )

Let’s break this down:

We first create line order instances from the Stripe session and send a confirmation email to the customer about their purchase.
We then send an email to all staff users telling them to check the admin panel.

You can now register the LineOrder model in the admin panel, so it’s accessible to staff users:

# ecommerce/admin.py
from django.contrib import admin
from ecommerce.models import LineOrder

# Register your models here.
admin.site.register(LineOrder)

When staff users log in to the admin page, they will now be able to check new orders and process them accordingly — in this case, pack and ship mugs to the customer!

Some Tips to Optimize Your Django Store

Here are some tips to further improve on the store you've built:

Write tests - you can see some examples in the GitHub repository.
If you have more products to sell, create a database model for them, and connect the LineOrder through a ForeignKey.
Configure email settings according to Django's email documentation. You can also use libraries such as django-post-office to manage your email templates and queues.
Once you deploy your website, create an actual webhook (not a local listener).
Take a look at the Stripe docs for alternatives to the checkout process we've outlined, including an embedded checkout form.

Wrapping Up

In this two-part series, we successfully built a one-product e-commerce site using Django, htmx, and Stripe. This guide has walked you through setting up your Django project, integrating htmx for seamless user interactions, and incorporating secure payments with Stripe.

We also covered how to handle order processing, including saving order information to your database, notifying staff users of new purchases, and sending confirmation emails to your customers. With these foundations, you can further customize and expand your e-commerce site to suit your specific needs.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Build a One-Product Shop With the Python Django Framework and Htmx

Roy Tomeij — 2024-08-28T00:00:00+00:00

This is the first of a two-part series using Django, htmx, and Stripe to create a one-product e-commerce website. In this part, we'll start our Django project and integrate it with htmx.

In the second part, we'll handle the orders with Stripe.

Let's get going!

Why Django, htmx, and Stripe?

We'll be using Django, htmx, and Stripe to create our website because:

Django is a Python web framework with a great ORM, templating, and routing system. It has a few features straight out of the box (like authentication) and multiple open source packages available online. We will use Django to build the website.
htmx is a JavaScript library that gives your website a modern feel just by using html attributes — you don’t actually have to write any JavaScript (although you can, of course). We will use htmx to give some interactivity to our page.
Stripe is a payments platform with a great API — it handles payments and credit card information. It also integrates nicely with Google and Apple Pay. We will use Stripe to facilitate product payments.

Here’s how the final product will work:

A user goes to our website and is able to see some information about our product, including its price and description.
Once the user clicks the “buy” button, they are redirected to Stripe’s checkout session.
If the payment is successful, they are redirected to our website again. We save their order information and send a confirmation email to the customer and all staff users to notify them of the recent purchase.

Now let's configure our Django project, create the initial views, and build the purchase form with htmx.

Configure Your Django Python Project

To set up our project, we need to create a virtual environment, activate it, and install the required packages. We can then create our Django project and Django app.

Create a Virtual Environment

Let's start by creating a virtual environment, so we can isolate our dependencies:

python -m venv .venv

Here's how we activate it on Linux/Mac:

source .venv/bin/activate

And on Windows:

.venv\Scripts\activate

Install the Required Packages

Within our activated virtual environment, we need a few packages to make this work:

pip install django stripe django-htmx python-dotenv

Here, we have installed:

Django
Stripe to work with the stripe API.
django-htmx, which provides some helper functionality to work with htmx.
python-dotenv to load environment variables from .env files.

Create the Django Project

In the same directory as our virtual environment, let's create a Django project called ecommerce_site:

django-admin startproject ecommerce_site .

In Django, it's good practice to have code organized by one or more "apps". Each app is a package that does something in particular. A project can have multiple apps, but for this simple shop, we can just have one app that will have most of the code — the views, forms, and models for our e-commerce platform. Let's create it and call it ecommerce:

python manage.py startapp ecommerce

And add this app to our INSTALLED_APPS in ecommerce_site/settings.py:

# ecommerce_site/settings.py

INSTALLED_APPS = [
    # ... the default django apps
    "ecommerce", # ⬅️ new
]

If you’re having trouble with this setup, check out the final product. At this stage, your file structure should look something like this:

ecommerce_site/
├── .venv/  # ⬅️ the virtual environment
├── ecommerce_site/ # ⬅️ the django project configuration
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── ecommerce/ # ⬅️ our app setup
│		├── templates/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── manage.py

Create the Templates

Now that we have our project configured, we need to create some base layouts. In the templates directory, add a base.html file — the template that all other templates will inherit from. Add htmx for user interaction, mvp.css for basic styling, and Django generated messages to the template:

<!-- ecommerce/templates/base.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>One-Product E-commerce Site</title>

    <!-- include htmx ⬇️ -->
    <script
      src="https://unpkg.com/htmx.org@1.9.11"
      integrity="sha384-0gxUXCCR8yv9FM2b+U3FDbsKthCI66oH5IA9fHppQq9DDMHuMauqq1ZHBpJxQ0J0"
      crossorigin="anonymous"
    ></script>

    <!-- include mvp.css ⬇️ -->
    <link rel="stylesheet" href="https://unpkg.com/mvp.css" />
  </head>
  <body hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}' hx-boost="true">
    <header>
      <h1>One-Product E-commerce Site</h1>
    </header>
    <main>
      <section>
        {% if messages %} {% for message in messages %}
        <p><mark>{{ message }}</mark></p>
        {% endfor %} {% endif %}
      </section>
      {% block content %} {% endblock %}
    </main>
  </body>
</html>

Create a home.html template in the same templates directory, for our home view. It should extend the base.html and just populate its content section.

<!-- ecommerce/templates/home.html -->

{% extends "base.html" %} {% block content %}
<section>{% include "product.html" %}</section>
{% endblock %}

In this template, we have included the product.html template. product.html will render some details about our product and a placeholder image. Let’s create it in the same templates directory:

<!-- ecommerce/templates/product.html -->
<form>
  <img src="https://picsum.photos/id/30/400/250" alt="mug" />
  <h3>mug<sup>on sale!</sup></h3>
  <p>mugs are great - you can drink coffee on them!</p>
  <p><strong>5€</strong></p>
  <button type="submit" id="submit-btn">Buy</button>
</form>

Create the Home View

In ecommerce/views.py, we'll create the view that will render the home template:

# ecommerce/views.py

from django.shortcuts import render

def home(request):
    return render(request, 'home.html')

And add it to the urlpatterns in ecommerce_site/urls.py:

# ecommerce_site/urls.py

from django.contrib import admin
from django.urls import path
from ecommerce import views # ⬅️ new

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", views.home, name="home"), # ⬅️ new
]

Now we can run the server with:

python manage.py runserver

If you jump over to http://127.0.0.1:8000 in your browser, you should see something like this:

It might feel like overkill to add a dedicated product.html template instead of just the product details in the home.html template, but product.html will be useful for the htmx integration.

Add the Form and Use Htmx

Great! We now have a view that looks good. However, it doesn’t do much yet. We'll add a form and set up the logic to process our product purchase. Here’s what we want to do:

Allow the user to select how many products (mugs) they want to buy.
When the user clicks “Buy”, make a POST request to a different view called purchase in the backend using hx-post.
In that view, validate the form and wait for 2 seconds to simulate the Stripe integration (we'll cover this in the second part of this guide).
Replace the form content with the purchase view response.
While the order is processing, show a loading message and disable the “Buy” button, so that the user doesn’t accidentally make multiple orders.

Let's go step by step.

1: Add a Quantity Order Form

Let’s first create and add a simple order form to our view allowing a user to select the number of mugs they want. In ecommerce/forms.py, add the following code:

# ecommerce/forms.py

from django import forms

class OrderForm(forms.Form):
        quantity = forms.IntegerField(min_value=1, max_value=10, initial=1)

In ecommerce/views.py, we can initialize the form in the home view:

# ecommerce/views.py

from ecommerce.forms import OrderForm # ⬅️ new

def home(request):
    form = OrderForm() # ⬅️ new - initialize the form
    return render(request, "home.html", {"form": form}) # ⬅️ new - pass the form to the template

And render it in the template:

<!-- ecommerce/templates/product.html -->

<form method="post">
  <!-- Same product details as before, hidden for simplicity -->

  <!-- render the form fields ⬇️ -->
  {{ form }}

  <!-- the same submit button as before ⬇️ -->
  <button type="submit" id="submit-btn">Buy</button>
</form>

2: Make a POST Request To a Different View

When the user clicks "Buy", we want to process the corresponding POST request in a dedicated view to separate the different logic of our application. We will use htmx to make this request. In the same ecommerce/templates/product.html template, let's extend the form attributes:

<!-- ecommerce/templates/product.html -->

<!-- add the hx-post html attribute ⬇️ -->
<form method="post" hx-post="{% url 'purchase' %}">
  <!-- Same product details as before, hidden for simplicity -->

  {{ form }}
  <button type="submit" id="submit-btn">Buy</button>
</form>

With this attribute, htmx will make a POST request to the purchase endpoint and stop the page from reloading completely. Now we just need to add the endpoint.

3: Create the Purchase View

The purchase view can be relatively simple for now:

# ecommerce/views.py
import time # ⬅️ new

# new purchase POST request view ⬇️
@require_POST
def purchase(request):
    form = OrderForm(request.POST)
    if form.is_valid():
        quantity = form.cleaned_data["quantity"]
        # TODO - add stripe integration to process the order
        # for now, just wait for 2 seconds to simulate the processing
        time.sleep(2)
    return render(request, "product.html", {"form": form})

In this view, we validate the form, extract the quantity from the cleaned data, and simulate Stripe order processing. In the end, we return the same template (product.html). We also need to add the view to the urlpatterns:

# ecommerce_site/urls.py

# ... same imports as before

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", views.home, name="home"),
    path("purchase", views.purchase, name="purchase"),  # ⬅️ new
]

We now need to tell htmx what to do with this response.

4: Replace the Form Content With the Purchase View Response

Htmx has a hx-swap attribute which replaces targeted content on the current page with a request's response. In our case, since the purchase view returns the same template, we want to swap its main element — the <form>:

<!-- ecommerce/templates/product.html -->

<!-- add the hx-swap html attribute ⬇️ -->
<form method="post" hx-post="{% url 'purchase' %}" hx-swap="outerHTML">
  <!-- Same product details as before, hidden for simplicity -->

  {{ form }}
  <button type="submit" id="submit-btn">Buy</button>
</form>

The value outerHTML will replace the entire target element with the response from the purchase view. By default, the htmx target is the same element making the call — in our case, the <form> element.

5: Tell Htmx What to Do When the Request Is Triggered

htmx provides some helper utilities for when a request is processing. In our case, we want to:

Disable the "Buy" button (to avoid a form re-submission) by using the hx-disabled-elt attribute.
Show a loading message on the screen by using the htmx-indicator class. This class changes its element's opacity when the request is triggered.

<!-- ecommerce/templates/product.html -->

<!-- add the hx-disabled-elt html attribute ⬇️ -->
<form
  method="post"
  hx-post="{% url 'purchase' %}"
  hx-swap="outerHTML"
  hx-disabled-elt="#submit-btn"
>
  <!-- Same product details as before, hidden for simplicity -->

  {{ form }}
  <button type="submit" id="submit-btn">Buy</button>

  <!-- add a loading message with the htmx-indicator class ⬇️ -->
  <p class="htmx-indicator"><small>Getting your mug ready...</small></p>
</form>

Once the customer clicks the "Buy" button, htmx will disable that same button (identifiable by its submit-btn id), and show a loading message by changing the opacity of the p.htmx-indicator element.

The Result

Jump over to the browser.

You can see the final result in this GitHub repository.

It doesn’t do much else yet. But, if form validation fails, errors will display directly on the form, without the need to refresh the entire page. If the form succeeds, we will process the Stripe order accordingly. We'll see how to do this in the second part of this guide.

Wrapping Up

We've now set up the basics for our one-product ecommerce site. We've configured our Django project, integrated it with htmx to give our site some interactivity, and set up a basic order form for our product.

In the second part of this guide, we'll handle orders with Stripe, save the results in our database, and notify users after a successful purchase.

Until then, happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Monitor the Performance of Your Python Django App with AppSignal

Roy Tomeij — 2024-07-31T00:00:00+00:00

When we observe a slow system, our first instinct might be to label it as failing. This presumption is widespread and highlights a fundamental truth: performance is synonymous with an application's maturity and readiness for production.

In web applications, where milliseconds can determine the success or failure of a user interaction, the stakes are incredibly high. Performance is not just a technical benchmark, but a cornerstone of user satisfaction and operational efficiency.

Performance encapsulates a system's responsiveness under varying workloads, quantified by metrics such as CPU and memory utilization, response times, scalability, and throughput.

In this article, we will explore how AppSignal can monitor and enhance the performance of Django applications.

Let's get started!

Django Performance Monitoring Essentials

Achieving optimal performance in Django applications involves a multifaceted approach. This means developing applications that run efficiently and maintain their efficiency as they scale. Key metrics are crucial in this process, providing tangible data to guide our optimization efforts. Let's explore some of these metrics.

Key Metrics for Performance Monitoring

Response Time: This is perhaps the most direct indicator of user experience. It measures the time taken for a user's request to be processed and the response sent back. In a Django application, factors like database queries, view processing, and middleware operations can influence response times.
Throughput: Throughput refers to the number of requests your application can handle within a given timeframe.
Error Rate: The frequency of errors (4xx and 5xx HTTP responses) can indicate code, database query, or server configuration issues. By monitoring error rates, you can quickly identify and fix problems that might otherwise degrade user experience.
Database Performance Metrics: These include the number of queries per request, query execution time, and the efficiency of database connections.
Handling Concurrent Users: When multiple users access your Django application simultaneously, it is critical to be able to serve all of them efficiently without delays.

What We Will Build

In this article, we'll construct a Django-based e-commerce store ready for high-traffic events, integrating AppSignal to monitor, optimize, and ensure it scales seamlessly under load. We'll also demonstrate how to enhance an existing application with AppSignal for improved performance (in this case, the Open edX learning management system).

Project Setup

Prerequisites

To follow along, you'll need:

Prepare the Project

Now let's create a directory for our project and clone it from GitHub. We'll install all the requirements and run a migration:

mkdir django-performance && cd django-performance
python3.12 -m venv venv
source venv/bin/activate
git clone -b main https://github.com/amirtds/mystore
cd mystore
python3.12 -m pip install -r requirements.txt
python3.12 manage.py migrate
python3.12 manage.py runserver

Now visit 127.0.0.1:8000. You should see something like this:

This Django application is a simple e-commerce store that provides product lists, details, and a checkout page for users. After successfully cloning and installing the app, use the Django createsuperuser management command to create a superuser.

Now let's create a couple of products in our application. First, we'll shell into our Django application by running:

python3.12 manage.py shell

Create 3 categories and 3 products:

from store.models import Category, Product

# Create categories
electronics = Category(name='Electronics', description='Gadgets and electronic devices.')
books = Category(name='Books', description='Read the world.')
clothing = Category(name='Clothing', description='Latest fashion and trends.')

# Save categories to the database
electronics.save()
books.save()
clothing.save()

# Now let's create new Products with slugs and image URLs
Product.objects.create(
    category=electronics,
    name='Smartphone',
    description='Latest model with high-end specs.',
    price=799.99,
    stock=30,
    available=True,
    slug='smartphone',
    image='products/iphone_14_pro_max.png'
)

Product.objects.create(
    category=books,
    name='Python Programming',
    description='Learn Python programming with this comprehensive guide.',
    price=39.99,
    stock=50,
    available=True,
    slug='python-programming',
    image='products/python_programming_book.png'
)

Product.objects.create(
    category=clothing,
    name='Jeans',
    description='Comfortable and stylish jeans for everyday wear.',
    price=49.99,
    stock=20,
    available=True,
    slug='jeans',
    image='products/jeans.png'
)

Now close the shell and run the server. You should see something like the following:

Install AppSignal

We'll install AppSignal and opentelemetry-instrumentation-django in our project.

Before installing these packages, log in to AppSignal using your credentials (you can sign up for a free 30-day trial). After selecting an organization, click on Add app at the top right of the navigation bar. Choose Python as your language and you'll receive a push-api-key.

Make sure your virtual environment is activated and run the following commands:

python3.12 -m pip install appsignal==1.2.1
python3.12 -m appsignal install --push-api-key [YOU-KEY]
python3.12 -m pip install opentelemetry-instrumentation-django==0.45b0

Provide the app name to the CLI prompt. After installation, you should see a new file called __appsignal__.py in your project.

Now let's create a new file called .env in the project root and add APPSIGNAL_PUSH_API_KEY=YOUR-KEY (remember to change the value to your actual key). Then, let's change the content of the __appsignal__.py file to the following:

# __appsignal__.py
import os
from appsignal import Appsignal

# Load environment variables from the .env file
from dotenv import load_dotenv
load_dotenv()

# Get APPSIGNAL_PUSH_API_KEY from environment
push_api_key = os.getenv('APPSIGNAL_PUSH_API_KEY')

appsignal = Appsignal(
    active=True,
    name="mystore",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
)

Next, update the manage.py file to read like this:

# manage.py
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys

# import appsignal
from __appsignal__ import appsignal # new line


def main():
    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mystore.settings")

    # Start Appsignal
    appsignal.start() # new line

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

We've imported AppSignal and started it using the configuration from __appsignal.py.

Please note that the changes we made to manage.py are for a development environment. In production, we should change wsgi.py or asgi.py. For more information, visit AppSignal's Django documentation.

Project Scenario: Optimizing for a New Year Sale and Monitoring Concurrent Users

As we approach the New Year sales on our Django-based e-commerce platform, we recall last year's challenges: increased traffic led to slow load times and even some downtime. This year, we aim to avoid these issues by thoroughly testing and optimizing our site beforehand. We'll use Locust to simulate user traffic and AppSignal to monitor our application's performance.

Creating a Locust Test for Simulated Traffic

First, we'll create a locustfile.py file that simulates simultaneous users navigating through critical parts of our site: the homepage, a product detail page, and the checkout page. This simulation helps us understand how our site performs under pressure.

Create the locustfile.py in the project root:

# locustfile.py
from locust import HttpUser, between, task

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)  # Users wait 1-3 seconds between tasks

    @task
    def index_page(self):
        self.client.get("/store/")

    @task(3)
    def view_product_detail(self):
        self.client.get("/store/product/smartphone/")

    @task(2)
    def view_checkout_page(self):
        self.client.get("/store/checkout/smartphone/")

In locustfile.py, users primarily visit the product detail page, followed by the checkout page, and occasionally return to the homepage. This pattern aims to mimic realistic user behavior during a sale.

Before running Locust, ensure you have a product with the smartphone slug in the Django app. If you don't, go to /admin/store/product/ and create one.

Defining Acceptable Response Times

Before we start, let's define what we consider an acceptable response time. For a smooth user experience, we aim for:

Homepage and product detail pages: under 1 second.
Checkout page: under 1.5 seconds (due to typically higher complexity).

These targets ensure users experience minimal delay, keeping their engagement high.

Conducting the Test and Monitoring Results

With our Locust test ready, we run it to simulate the 500 users and observe the results in real time. Here's how:

Start the Locust test by running locust -f locustfile.py in your terminal, then open http://localhost:8089 to set up and start the simulation. Set the Number of Users to 500 and set the host to http://127.0.0.1:8000
Monitor performance in both Locust's web interface and AppSignal. Locust shows us request rates and response times, while AppSignal provides deeper insights into our Django app's behavior under load.

After running Locust, you can find information about the load test in its dashboard:

Now, go to your application page in AppSignal. Under the Performance section, click on Actions and you should see something like this:

Mean: This is the average response time for all the requests made to a particular endpoint. It provides a general idea of how long it takes for the server to respond. In our context, any mean response time greater than 1 second could be considered a red flag, indicating that our application's performance might not meet user expectations for speed.
90th Percentile: This is the response time at the 90th percentile. For example, for GET store/, we have 7 ms, which means 90% of requests are completed in 7 ms or less.
Throughput: The number of requests handled per second.

Now let's click on the Graphs under Performance:

We need to prepare our site for the New Year sales, as response times might exceed our targets. Here's a simplified plan:

Database Queries: Slow queries often cause performance issues.
Static Assets: Ensure static assets are properly cached. Use a CDN for better delivery speeds.
Application Resources: Sometimes, the solution is as straightforward as adding more RAM and CPUs.

Database Operations

Understanding Database Performance Impact

When it comes to web applications, one of the most common sources of slow performance is database queries. Every time a user performs an action that requires data retrieval or manipulation, a query is made to the database.

If these queries are not well-optimized, they can take a considerable amount of time to execute, leading to a sluggish user experience. That's why it's crucial to monitor and optimize our queries, ensuring they're efficient and don't become the bottleneck in our application's performance.

Instrumentation and Spans

Before diving into the implementation, let's clarify two key concepts in performance monitoring:

Instrumentation
Spans

Instrumentation is the process of augmenting code to measure its performance and behavior during execution. Think of it like fitting your car with a dashboard that tells you not just the speed, but also the engine's performance, fuel efficiency, and other diagnostics while you drive.

Spans, on the other hand, are the specific segments of time measured by instrumentation. In our car analogy, a span would be the time taken for a specific part of your journey, like from your home to the highway. In the context of web applications, a span could represent the time taken to execute a database query, process a request, or complete any other discrete operation.

Instrumentation helps us create a series of spans that together form a detailed timeline of how a request is handled. This timeline is invaluable for pinpointing where delays occur and understanding the overall flow of a request through our system.

Implementing Instrumentation in Our Code

With our PurchaseProductView, we're particularly interested in the database interactions that create customer records and process purchases. By adding instrumentation to this view, we'll be able to measure these interactions and get actionable data on their performance.

Here's how we integrate AppSignal's custom instrumentation into our Django view:

# store/views.py
# Import OpenTelemetry's trace module for creating custom spans
from opentelemetry import trace
# Import AppSignal's set_root_name for customizing the trace name
from appsignal import set_root_name

# Inside the PurchaseProductView
def post(self, request, *args, **kwargs):
    # Initialize the tracer for this view
    tracer = trace.get_tracer(__name__)

    # Start a new span for the view using 'with' statement
    with tracer.start_as_current_span("PurchaseProductView"):
        # Customize the name of the trace to be more descriptive
        set_root_name("POST /store/purchase/<slug>")

        # ... existing code to handle the purchase ...

        # Start another span to monitor the database query performance
        with tracer.start_as_current_span("Database Query - Retrieve or Create Customer"):
            # ... code to retrieve or create a customer ...

            # Yet another span to monitor the purchase record creation
            with tracer.start_as_current_span("Database Query - Create Purchase Record"):
                # ... code to create a purchase record ...

See the full code of the view after the modification.

In this updated view, custom instrumentation is added to measure the performance of database queries when retrieving or creating a customer and creating a purchase record.

Now, after purchasing a product in the Slow events section of the Performance dashboard, you should see the purchase event, its performance, and how long it takes to run the query.

purchase is the event we added to our view.

Using AppSignal with an Existing Django App

In this section, we are going to see how we can integrate AppSignal with Open edX, an open-source learning management system based on Python and Django.

Monitoring the performance of learning platforms like Open edX is highly important, since a slow experience directly impacts students' engagement with learning materials and can have a negative impact (for example, a high number of users might decide not to continue with a course).

Integrate AppSignal

Here, we can follow similar steps as the Project Setup section. However, for Open edX, we will follow Production Setup and initiate AppSignal in wsgi.py. Check out this commit to install and integrate AppSignal with Open edX.

Monitor Open edX Performance

Now we'll interact with our platform and see the performance result in the dashboard.

Let's register a user, log in, enroll them in multiple courses, and interact with the course content.

Actions

Going to Actions, let's order the actions based on their mean time and find slow events:

As we can see, for 3 events (out of the 34 events we tested) the response time is higher than 1 second.

Host Metrics

Host Metrics in AppSignal show resource usage:

Our system isn't under heavy load — the load average is 0.03 — but memory usage is high.

We can also add a trigger to get a notification when resource usage reaches a specific condition. For example, we can set a trigger for when memory usage gets higher than 80% to get notified and prevent outages.

When we hit the condition, you should get a notification like the following:

Celery Tasks Monitoring

In Open edX, we use Celery for async and long-running tasks like certificate generation, grading, and bulk email functionality. Depending on the task and the number of users, some of these tasks can run for a long time and cause performance issues in our platform.

For example, if thousands of users are enrolled in a course and we need to rescore them, this task can take a while. We might get complaints from users that their grade is not reflected in the dashboard because the task is still running. Having information on Celery tasks, their runtime, and their resource usage gives us important insights and possible points of improvement for our application.

Let's use AppSignal to trace our Celery tasks in Open edX and see the result in the dashboard. First, make sure the necessary requirements are installed. Next, let's set our tasks to trace Celery performance like in this commit.

Now, let's run a couple of tasks in the Open edX dashboard to reset attempts and rescore learners' submissions:

We'll go to the Performance dashboard in AppSignal -> Slow events and we will see something like:

By clicking on Celery, we'll see all the tasks that have run on Open edX:

This is great information that helps us see if our tasks are running longer than expected, so we can address any possible performance bottlenecks.

And that's it!

Wrapping Up

In this article, we saw how AppSignal can give us insights into our Django app's performance.

We monitored a simple e-commerce Django application, including metrics like concurrent users, database queries, and response time.

As a case study, we integrated AppSignal with Open edX, revealing how performance monitoring can be instrumental in enhancing the user experience, particularly for students using the platform.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Monitor the Performance of your Python FastAPI App with AppSignal

Roy Tomeij — 2024-07-10T00:00:00+00:00

While building an app with FastAPI can be reasonably straightforward, deploying and operating it might be more challenging.

The whole user experience can be ruined by unexpected errors, slow responses, or even worse — downtime.

AppSignal is a great tool of choice for efficiently tracking your FastAPI app's performance. It allows you to easily monitor average/95th percentile/90th percentile response times, error rates, throughput, and much more. Useful charts are available out of the box. Let's see it in action!

What Can You Do with Performance Monitoring?

With performance monitoring, we can track app response times, throughput, error rates, CPU consumption, memory usage, etc. Changes in these metrics can indicate something is not quite right, and we should investigate.

For example, if response times are monotonically increasing on a specific endpoint, we can investigate what's causing it. It might be inefficient code, a slow database query, a slow external API call, or something else.

In such cases, you can intervene before a gateway timeout occurs, for example, and your users start complaining.

Setting Up Our FastAPI Python Project and Configuring AppSignal

Let's use an app I've already prepared.

First, clone the repository from GitHub:

$ git clone git@github.com:jangia/fastapi_performance_with_appsignal.git
$ cd fastapi_performance_with_appsignal

Second, create a virtual environment and install the dependencies:

$ python3.12 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Note: For things to work, you need the following packages installed: opentelemetry-instrumentation-fastapi and appsignal.

Third, set the AppSignal environment variables:

$ export APPSIGNAL_PUSH_API_KEY=<your_appsignal_push_api_key>
$ export APPSIGNAL_REVISION=main

Note: You can read more about configuring AppSignal for FastAPI in Track Errors in FastAPI for Python with AppSignal.

Use environment variables to configure AppSignal in the __appsignal__.py file:

import os

from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="fastapi_performance_with_appsignal",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
    revision=os.getenv("APPSIGNAL_REVISION"),
    enable_host_metrics=True,
)

The FastAPI app looks like this:

import json
import random
import time

import requests

from appsignal import set_category, set_sql_body, set_body
from fastapi import FastAPI, Depends
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from sqlalchemy.orm import Session

from __appsignal__ import appsignal
from models import SessionLocal, Task

appsignal.start()

tracer = trace.get_tracer(__name__)

app = FastAPI(
    title="FastAPI with AppSignal",
)


def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()


@app.get("/hello-world")
def hello_world():
    time.sleep(random.random())
    return {"message": "Hello World"}

@app.get("/error")
def hello_world():
    raise Exception("Something went wrong. Oops!")


@app.get("/slow-external-api")
def slow_external_api():
    api_url = "http://docs.appsignal.com/"
    with tracer.start_as_current_span("Call External API"):
        set_category("external_api.http")
        set_body(json.dumps({"url": api_url}))
        response = requests.get(api_url)

    return {"message": "External API successfully called!"}


@app.get("/slow-query")
def slow_query(db: Session = Depends(get_db)):

    with tracer.start_as_current_span("List tasks"):
        query = db.query(Task)
        tasks = query.all()
        set_category("tasks.sql")
        set_sql_body(str(query))
    return {
        "tasks": [
            {"id": task.id, "title": task.title, "status": task.status}
            for task in tasks
        ],
    }


FastAPIInstrumentor().instrument_app(app)

We'll use a few endpoints to demonstrate how to monitor performance.

Monitor Response Times and Throughput in AppSignal

With our dependencies installed and environment variables set, we can start the app:

(venv)$ uvicorn main:app --reload

Once the app is up and running, we can use the call_api.py script to send requests to the app endpoints in parallel:

(venv)$ python call_api.py

This script will call every endpoint we have 20 times in parallel using asyncio.

import asyncio
from aiohttp import ClientSession


async def call_api(url: str):
    async with ClientSession() as session:
        async with session.get(url) as response:
            response = await response.text()
            print(response)


async def main():
    base_url = "http://localhost:8000"
    endpoints = ["slow-query", "slow-external-api", "hello-world", "error"]
    async with asyncio.TaskGroup() as group:
        for endpoint in endpoints:
            url = f"{base_url}/{endpoint}"
            for i in range(20):
                group.create_task(call_api(url))

asyncio.run(main())

Once the script completes, go to the AppSignal dashboard and select your application. Once on the application dashboard, choose the Performance tab -> Graphs. You'll see two charts — Response Times and Throughput:

The Response Times chart shows the average response time, 95th percentile response time, and 90th percentile response time across all endpoints inside your application.

The Throughput chart shows the number of processed requests per minute across all endpoints inside your application.

Looking at these charts is a great way to get a quick overview of how your application is performing, but it's not enough. Fortunately, AppSignal provides a way to drill into the details — you can see the same charts per endpoint.

Digging Deeper

To do that, click on Actions inside Performance:

You'll see a list of all endpoints inside your app that were called within the specified time range.

For each endpoint, you can see the average response time, 95th percentile response time, and 90th percentile response time. You can go even deeper by clicking on the endpoint name. That will take you to the endpoint details page.

You'll see the same charts as before. This time, they are for the selected endpoint only.

You'll see another thing on the endpoint details page — errors and the error rate for the selected endpoint:

Looking at these charts, you can quickly see whether:

Response times are increasing/decreasing/stable
Throughput is increasing/decreasing/stable
Error rate is increasing/decreasing/stable

If any of these metrics show a trend in the wrong direction, you can investigate and fix them before they become a problem for your users.

Setting Alerts

While this is all great and useful, you must go to the AppSignal dashboard to see how your app is performing. This way, you might still not react quickly enough when something goes wrong.

To overcome this, you can set an alert that triggers whenever a request takes longer than the specified threshold.

Go to the details of a /slow-query endpoint and click on View Incident.

After that, set alerts on the right side, with 5 seconds as the threshold, and select Every occurrence as Alerting:

Run the script that calls the endpoints one more time:

(venv)$ python call_api.py

You should receive an email from AppSignal warning you about the request/s that take/took too long to process. This means you don't need to constantly check the AppSignal dashboard to see how your app is performing. You'll be notified straight away if something strange happens.

Monitor Database Queries

Monitoring response times and throughput is very useful, but it only tells us a little about what's causing a problem.

As mentioned, multiple things can cause slow response times.

One is slow database queries: we'll also want to monitor them.

Let's take a look at the /slow-query endpoint:

# ... other code
tracer = trace.get_tracer(__name__)
# ... other code


@app.get("/slow-query")
def slow_query(db: Session = Depends(get_db)):

    with tracer.start_as_current_span("List tasks"):
        query = db.query(Task)
        tasks = query.all()
        set_category("tasks.sql")
        set_sql_body(str(query))
    return {
        "tasks": [
            {"id": task.id, "title": task.title, "status": task.status}
            for task in tasks
        ],
    }

Here, we're using SQLAlchemy to query the database for all tasks. To track queries inside FastAPI, we need to utilize custom instrumentation.

We can do that by executing the query inside the with tracer.start_as_current_span("List tasks"): block. We'll create a "List tasks" span. You'll see it inside the AppSignal dashboard.

We execute the query inside the span. We need to set a category for AppSignal to recognize that this is a database query measurement using set_category("tasks.sql").

The category name must end with one of the following suffixes to be recognized as a database query:

*.active_record
*.ecto
*.elasticsearch
*.knex
*.mongodb
*.mysql
*.postgres
*.psycopg2
*.redis
*.sequel
*.sql

We also add a query string to the span body — set_sql_body(str(query)). This way, we can see the executed query inside the AppSignal dashboard.

Note: str(query) returns a query with placeholders. If you want to see a query with values, you can use str(query.statement.compile(compile_kwargs={"literal_binds": True})). But be careful not to send any sensitive data this way.

Note: A span is a single step in the execution flow.

Since you've already executed the script that calls the endpoints, you should see the /slow-query endpoint query inside the AppSignal dashboard.

Go to Slow queries under Performance:

You can click on the query name tasks.sql to see the query details. There you'll find:

The query itself (as we sent it, with set_sql_body)
Response times chart
Throughput chart

Looking at the charts, you can see the query's performance trend. If the response times are monotonically increasing, you can investigate why and fix the issue.

Hint: some possible causes for slow queries are:

Queries are not using indexes (e.g., there's a missing index or filtering is only set on non-indexed columns).
Queries are loading all the data from a database (e.g., there's missing pagination or all the data for rows is being loaded despite this being unnecessary).
N + 1 queries (e.g., querying for all users and then querying each user's tasks to satisfy a single request).
Inefficient query planning (e.g., max set on an empty result can take an abnormally long time in PostgreSQL).

Monitor Slow External API Calls

External API calls can also cause slow response times. Since external APIs are outside your control, you just have to hope for the best.

In reality, external APIs often cause problems due to being unresponsive or even down entirely. That's why you should monitor those calls as well.

As with database queries, AppSignal has got you covered.

Let's take a look at the /slow-external-api endpoint:

@app.get("/slow-external-api")
def slow_external_api():
    api_url = "http://docs.appsignal.com/"
    with tracer.start_as_current_span("Call External API"):
        set_category("external_api.http")
        set_body(json.dumps({"url": api_url}))
        requests.get(api_url)

    return {"message": "External API successfully called!"}

Like with database queries, we need to use custom instrumentation to track external API calls. We can do that by executing the query inside the with tracer.start_as_current_span("Call External API") block.

We set the span name to "Call External API". You'll see it inside the AppSignal dashboard.

Inside the span, we execute the external API call. Set the correct category for AppSignal to recognize that this is an external API call measurement — set_category("external_api.http").

The category name must end with one of the following suffixes to be recognized as an external API call:

*.faraday
*.grpc
*.http
*.http_rb
*.net_http
*.excon
*.request
*.requests
*.service
*.finch
*.tesla
*.fetch

We also add a called URL to the span body — set_body(json.dumps({"url": api_url})). This lets us see which URL was called inside the AppSignal dashboard. You can add more details to the span's body by extending the dictionary you're sending to set_body. (For example, you can also add a request or response body).

Note: Ensure you don't send sensitive data inside the span's body.

Since you've already executed the script that calls the endpoints, you should see the /slow-external-api endpoint query inside the AppSignal dashboard.

Go to Slow queries under Performance:

You can click on the API call name external_api.http to see the details. There you'll find:

The URL that was called (as we sent it with set_body)
Response times chart
Throughput chart

Looking at the charts, you can see how the API call performs over time. You can decide whether to increase the timeout for the API call or change API usage (e.g., use a smaller page size or utilize a different API).

Monitor Host Metrics in AppSignal for FastAPI

Last but not least, you can monitor host metrics. AppSignal can track CPU, memory, disk, and network usage. You can enable that by setting enable_host_metrics=True inside the __appsignal__.py file:

import os

from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="fastapi_performance_with_appsignal",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
    revision="main",
    enable_host_metrics=True,  # THIS
)

So stop uvicorn, and let's run our API with docker-compose:

$ docker-compose up -d --build

Once again, run the script that calls the endpoints:

(venv)$ python call_api.py

Note: macOS/OSX is not supported. That's why we're using Docker. See AppSignal's official docs for more info.

Once the script completes, go to the AppSignal dashboard under Performance -> Host metrics — you'll see the list of hosts that are running your app:

Click on the hostname to see the details. There, you'll find these charts:

Load Average
CPU Usage
Memory Usage
Swap Usage
Disk I/O Read
Disk I/O Write
Disk Usage
Network Traffic Received
Network Traffic Transmitted

These metrics can help you determine whether your hosts have enough resources to handle the load. You can also see whether the load is roughly equally spread across the hosts.

If you see anything that concerns you, you can investigate further and take action before it becomes a problem for your users.

And that's it!

Wrapping Up

In this post, we've seen how to monitor the performance of FastAPI applications using AppSignal.

Monitoring performance means that we can intervene before things go south and our users start complaining.

To be fully in control of our app, we should combine performance monitoring with error tracking.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Deploy a Python FastAPI Application to Render

Roy Tomeij — 2024-06-26T00:00:00+00:00

In the world of Python frameworks, FastAPI is the new kid on the block and a great choice for building APIs. Equally, Render is a good option for developers who want to quickly test their applications in a production environment for free.

In this post, we'll run through how to deploy a FastAPI app to Render. First, though, let's explore why FastAPI and Render are often chosen by developers.

Why FastAPI?

FastAPI is a high-performance microframework used primarily for building APIs (the clue is in the name). As such, FastAPI offers several advantages over older, better-known frameworks such as Django and Flask.

The first and most obvious advantage of FastAPI is that it was built with scalability and performance in mind.

For example, FastAPI is based on an asynchronous ASGI server rather than the older WSGI used by Django and other frameworks. ASGI servers can handle multiple requests simultaneously (concurrently). This is often seen as a better option for apps that need to handle high levels of user traffic.

FastAPI also makes it easier for developers to write asynchronous code by simply using the async keyword when defining asynchronous functions.

But FastAPI's shiny "newness" is also its main drawback. Because it was only introduced in 2018, FastAPI has a much smaller community and fewer learning resources (such as coding tutorials) compared to the more established frameworks. This is something to bear in mind when choosing FastAPI for your next project.

Why Render?

Render is a great option for developers who want to quickly test their applications in a production environment for free.

With Render, you don't even have to enter your credit card details to gain access to the free tier. So unlike other cloud services, there is no way to rack up charges by mistake.

As we will see in the section below, Render also provides an excellent overall developer experience with a clean, very easy-to-use UI and smooth integration with Git and GitHub. An added bonus is that any app you host on Render gets a free TLS certificate right out of the box.

The downside with Render's free tier is that it can take a long time for deployments to complete. Once this starts to become an issue, you might want to consider upgrading to one of the paid plans.

Create a FastAPI Demo Application

You can view, download, or clone the full code used in this article.

The first thing we need to do is create an empty directory where our project will live. Let's call it fastapi-render-appsignal.

Run the following commands in your terminal:

mkdir fastapi-render-appsignal
cd flask-heroku-appsignal
touch main.py

The entry point for the project will be main.py.

Next, we need to make our project a git repository by running the git initialize command in the root of the directory:

git init

Also, create a requirements.txt file in the project's root directory. Add these two lines to the file:

fastapi
uvicorn[standard]

Then run:

pip install -r requirements.txt

If the installation goes smoothly, you should see the following output in the terminal:

Installing collected packages: websockets, uvloop, typing-extensions, sniffio, pyyaml, python-dotenv, idna, httptools, h11, exceptiongroup, click, annotated-types, uvicorn, pydantic-core, anyio, watchfiles, starlette, pydantic, fastapi
Successfully installed annotated-types-0.6.0 anyio-4.2.0 click-8.1.7 exceptiongroup-1.2.0 fastapi-0.109.0 h11-0.14.0 httptools-0.6.1 idna-3.6 pydantic-2.5.3 pydantic-core-2.14.6 python-dotenv-1.0.1 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.35.1 typing-extensions-4.9.0 uvicorn-0.27.0 uvloop-0.19.0 watchfiles-0.21.0 websockets-12.0

Now, in main.py, copy and paste the following barebones boilerplate code:

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

Let's go line-by-line and explain what is being done in this short code snippet.

After the import statements, we create an instance of the FastAPI class and assign it to the variable app.
@app.get("/") is a Python decorator that tells FastAPI to create an API endpoint via the HTTP GET method at the root URL /.
async def root() defines an asynchronous function named root. We are using the async keyword here because FastAPI is built to work with asynchronous I/O operations. This means it can handle concurrent requests, which is better for performance.
In the body of the function, we return a Python dictionary which FastAPI helpfully converts to JSON for us. So, when the user goes to the root / endpoint, our API will automatically respond with a JSON object.

We can now run our project using the server package Uvicorn (that we installed earlier) with the following command:

uvicorn main:app --reload

If we open our browser at http://127.0.0.1:8000, we will just see the following plain JSON response:

{ "message": "Hello World" }

But if we add /docs to the end of the URL, we can take advantage of FastAPI's in-built interactive API documentation capabilities.

The documentation UI in the screenshot below is provided by Swagger UI:

You can also try another documentation UI style that's automatically included with FastAPI.

Enter the URL http://127.0.0.1:8000/redoc in your browser.

This one is provided by ReDoc:

Deploy on Render

This part assumes you have already created a repository for this project on GitHub. If you need instructions on how to do this, check out GitHub’s official docs.

One of Render’s advantages is that it provides an easy and intuitive way to connect to your GitHub repository.

First, go to render.com and create a new account with Render (you can use your GitHub credentials to speed things up).

Then click on the New button and select the option to create a new web service.

After linking your GitHub account, you should see the screen below, which provides a list of GitHub repos to connect to Render. Let's select our fastapi-render-appsignal example project.

This will then take us to the main configuration screen. Under region, simply select the region closest to your location (since I’m based in the UK, I will choose Frankfurt):

Further down the screen, we have a few more important options.

The build command will use the requirements.txt file we created earlier in the project to install all the packages from our project on Render’s remote server.

Under start command, we will use Uvicorn again in our production environment and tell Render to start the application using main.py in the root of our project.

Note: You must explicitly tell Render which host and port to use (unlike when using other servers like gunicorn). Unfortunately, this is not mentioned in any of the official Render documentation but I found the solution in the Render community forum and in the post 'Deploying FastAPI application to Render'.

Next, choose the free option under the list of instance types:

Lastly, under environment variables, specify the Python version as 3.10.7. Ensure this matches the same version you are using in your local project.

With our web service configuration done, now we simply click the Create Web Service button and watch the project get deployed:

Remember, this will take a bit of time since we are using the free plan.

If everything goes smoothly at the end of the deployment process, we should see a final message in Render's logs saying: "Your service is live".

Now we can click on the public link Render has generated for us and see our Hello World FastAPI example, including the automatic documentation links at /docs and /redoc:

Installing the AppSignal Dashboard for FastAPI and Monitoring Deploys

It's great that we now have a basic FastAPI app running and deployed to Render, but what if something goes wrong? Even with the best manual and automated testing, the reality is that errors and bugs happen in software development. The key thing is to get alerted as soon as errors happen and to quickly identify where the problem is in the code.

In this section of the tutorial, we are going to specifically look at how to enable AppSignal to monitor your deployments. This part of the setup is crucial for you to see which errors are associated with a specific deployment. This will allow you to diagnose and fix problems in your app quickly and efficiently.

First, sign up for an AppSignal account (you can do a 30-day free trial, no credit card details required). Next, add appsignal and opentelemetry-instrumentation-fastapi to your requirements.txt file:

# requirements.txt
appsignal
opentelemetry-instrumentation-fastapi

Then run:

pip install -r requirements.txt

Now we need to create an appsignal.py configuration file and add the following code:

import os
import subprocess
from appsignal import Appsignal
from dotenv import load_dotenv
load_dotenv()

revision = None
try:
    revision = subprocess.check_output(
        "git log --pretty=format:'%h' -n 1", shell=True
    ).strip()
except subprocess.CalledProcessError:
  pass


appsignal = Appsignal(
    active=True,
    name="fastapi-render-appsignal",
    push_api_key=os.getenv("APPSIGNAL_API_KEY"),
    revision=revision,
)

In the code above, we initialize the Appsignal object with a few different parameters, including:

Setting active to True to enable monitoring.
Providing the name for our application.
Putting the API key in an environment variable.

But the key parameter to enable deploy tracking is the revision parameter. Inside the try-except block (above the AppSignal initialization code), we fetch the latest commit hash so AppSignal can keep track of our deploys.

Lastly, we need to update main.py to finish setting up AppSignal in our FastAPI app:

# main.py
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor #new
import appsignal #new

appsignal.start() #new

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

@app.get("/items/{item_id}")
async def get_item(item_id: int):
    return {"item_id": item_id}

FastAPIInstrumentor().instrument_app(app) #new

When we check out our AppSignal dashboard, the first thing we will see is the Getting Started screen which will provide some suggestions on additional configuration steps to complete:

But here we'll just focus on ensuring that AppSignal can track our deploys.

Go to the Organization page in the AppSignal dashboard and then click on Organization settings to activate AppSignal's integration with GitHub. Check out AppSignal's official docs on linking your repo for step-by-step instructions.

With the GitHub integration set up and your repo linked, all you need to do now is trigger a deployment in Render for AppSignal to start monitoring your deploys.

In our case, we simply need to push to our main branch:

git push origin main

If everything is set up correctly, we should see AppSignal tracking our deploys and any errors associated with each deploy commit hash in the Deploys section of the dashboard:

And that's it!

Wrapping Up

In this guide, we covered a lot of ground! First, we briefly introduced some of the advantages and disadvantages of choosing FastAPI and Render.

Then, we created a simple FastAPI app, deployed it to Render, and ended by monitoring those deploys with AppSignal.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Track Errors in Your Python Flask Application with AppSignal

Roy Tomeij — 2024-05-29T00:00:00+00:00

In this article, we'll look at how to track errors in a Flask application using AppSignal.

We'll first bootstrap a Flask project, and install and configure AppSignal. Then, we'll introduce some faulty code and demonstrate how to track and resolve errors using AppSignal's Errors dashboard.

Let's get started!

Prerequisites

Before diving into the article, ensure you have:

Python 3.8+ installed on your local machine
An AppSignal-supported operating system
An AppSignal account (you can start a free 30-day trial)
Fundamental Flask knowledge

Project Setup

To demonstrate how AppSignal error tracking works, we'll create a simple TODO app. The app will provide a RESTful API that supports CRUD operations. Initially, it will contain some faulty code, which we'll address later.

I recommend you first follow along with this exact project since the article is tailored to it. After the article, you'll, of course, be able to integrate AppSignal into your own Flask projects.

Start by bootstrapping a Flask project:

Create and activate a virtual environment
Use pip to install the latest version of Flask
Start the development server

If you get stuck, refer to the Flask Installation guide.

Note: The source code for this project can be found in the appsignal-flask-error-tracking GitHub repo.

Install AppSignal for Flask

To add AppSignal to your Flask project, follow the AppSignal documentation:

Ensure everything works by starting the development server:

(venv)$ flask run

Your app should automatically send a demo error to AppSignal. From now on, all your app errors will be forwarded to AppSignal.

If you get an error saying Failed to find Flask application, you most likely imported Flask before starting the AppSignal client. As mentioned in the docs, AppSignal has to be imported and started at the top of app.py.

Flask for Python App Logic

Moving along, let's implement the web app logic.

Flask-SQLAlchemy for the Database

We'll use the Flask-SQLAlchemy package to manage the database. This package provides SQLAlchemy support to Flask projects. That includes the Python SQL toolkit and the ORM.

First, install it via pip:

(venv)$ pip install Flask-SQLAlchemy

Then initialize the database and Flask:

# app.py

db = SQLAlchemy()
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///default.db"
db.init_app(app)

Don't forget about the import:

from flask_sqlalchemy import SQLAlchemy

Next, create the Task database model:

# app.py

class Task(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(128), nullable=False)
    description = db.Column(db.Text(512), nullable=True)

    created_at = db.Column(db.DateTime, default=db.func.now())
    updated_at = db.Column(db.DateTime, default=db.func.now(), onupdate=db.func.now())

    is_done = db.Column(db.Boolean, default=False)

    def as_dict(self):
        return {c.name: getattr(self, c.name) for c in self.__table__.columns}

    def __repr__(self):
        return f"<Task {self.id}>"

Each Task will have a name, an optional description, an is_done field, and some administrative data. To serialize the Task, we'll use its as_dict() method.

Since Flask-SQLAlchemy doesn't automatically create the database and its structure, we must do it ourselves. To handle that, we'll create a simple Python script.

Create an init_db.py file in the project root with the following content:

# init_db.py

from app import Task
from app import db, app

with app.app_context():
    db.create_all()

    if Task.query.count() == 0:
        tasks = [
            Task(
                name="Deploy App",
                description="Deploy the Flask app to the cloud.",
                is_done=False,
            ),
            Task(
                name="Optimize DB",
                description="Optimize the database access layer.",
                is_done=False,
            ),
            Task(
                name="Install AppSignal",
                description="Install AppSignal to track errors.",
                is_done=False,
            ),
        ]

        for task in tasks:
            db.session.add(task)

        db.session.commit()

What's Happening Here?

This script performs the following:

Fetches Flask's app instance.
Creates the database and its structure via db.create_all().
Populates the database with three sample tasks.
Commits all the changes to the database via db.session.commit().

Defining Views

Define the views in app.py like so:

# app.py

@app.route("/")
def list_view():
    tasks = Task.query.all()
    return jsonify([task.as_dict() for task in tasks])


@app.route("/<int:task_id>", methods=["GET"])
def detail_view(task_id):
    task = db.get_or_404(Task, task_id)
    return jsonify(task.as_dict())


@app.route("/create", methods=["POST"])
def create_view():
    name = request.form.get("name", type=str)
    description = request.form.get("description", type=str)

    task = Task(name=name, description=description)
    db.session.add(task)
    db.session.commit()

    return jsonify(task.as_dict()), 201


@app.route("/toggle-done/<int:task_id>", methods=["PATCH"])
def toggle_done_view(task_id):
    task = db.get_or_404(Task, task_id)

    task.is_done = not task.is_done
    db.session.commit()

    return jsonify(task.as_dict())


@app.route("/delete/<int:task_id>", methods=["DELETE"])
def delete_view(task_id):
    task = db.get_or_404(Task, task_id)

    db.session.delete(task)
    db.session.commit()

    return jsonify({}), 204


@app.route("/statistics", methods=["GET"])
def statistics_view():
    done_tasks_count = Task.query.filter_by(is_done=True).count()
    undone_tasks_count = Task.query.filter_by(is_done=False).count()
    done_percentage = done_tasks_count / (done_tasks_count + undone_tasks_count) * 100

    return jsonify({
        "done_tasks_count": done_tasks_count,
        "undone_tasks_count": undone_tasks_count,
        "done_percentage": done_percentage,
    })

Don't forget about the import:

from flask import jsonify, request

What's Happening Here?

We define six API endpoints.
The list_view() fetches all the tasks, serializes and returns them.
The detail_view() fetches a specific task, serializes and returns it.
The create_view() creates a new task from the provided data.
toggle_done_view() toggles the task's is_done property.
The delete_view() deletes a specific task.
The statistics_view() calculates general app statistics.

Great, we've successfully created a simple TODO web app!

Test Your Python Flask App's Errors with AppSignal

During the development of our web app, we intentionally left in some faulty code. We'll now trigger these bugs to see what happens when an error occurs.

Before proceeding, ensure your Flask development server is running:

(venv)$ flask run --debug

Your API should be accessible at http://localhost:5000/.

AppSignal should, of course, be employed when your application is in production rather than during development, as shown in this article.

Error 1: `OperationalError`

To trigger the first error, request the task list:

$ curl --location 'localhost:5000/'

This will return an Internal Server Error. Let's use AppSignal to figure out what went wrong.

Open your favorite web browser and navigate to your AppSignal dashboard. Select your organization and then your application. Lastly, choose "Errors > Issue list" on the sidebar:

You'll see that an OperationalError was reported. Click on it to inspect it:

The error detail page will display the error message, backtrace, state, trends, and so on.

We can figure out what went wrong just by looking at the error message. no such table: task tells us that we forgot to initialize the database.

To fix that, run the previously created script:

(venv)$ python init_db.py

Retest the app and mark the issue as "Closed" once you've verified everything works.

Error 2: `IntegrityError`

Let's trigger the next error by trying to create a task without a name:

$ curl --location 'localhost:5000/create' \
       --form 'description="Test the web application."'

Open the AppSignal dashboard and navigate to the IntegrityError's details.

Now instead of just checking the error message, select "Samples" in the navigation:

A sample refers to a recorded instance of a specific error. Select the first sample.

By checking the backtrace, we can see exactly what line caused the error. As you can see, the error happened in app.py on line 53 when we tried saving the task to the database.

To fix it, provide a default when assigning the name variable:

# app.py

@app.route("/create", methods=["POST"])
def create_view():
    name = request.form.get("name", type=str, default="Unnamed Task")  # new
    description = request.form.get("description", type=str, default="")

    task = Task(name=name, description=description)
    db.session.add(task)
    db.session.commit()

    return jsonify(task.as_dict()), 201

Error 3: `ZeroDivisionError`

Now we'll delete all tasks and then calculate the statistics by running the following commands:

$ curl --location --request DELETE 'localhost:5000/delete/1'
$ curl --location --request DELETE 'localhost:5000/delete/2'
$ curl --location --request DELETE 'localhost:5000/delete/3'
$ curl --location --request DELETE 'localhost:5000/delete/4'
$ curl --location --request DELETE 'localhost:5000/delete/5'
$ curl --location 'localhost:5000/statistics'

As expected, a ZeroDivisonError is raised. To track the error, follow the same approach as described in the previous section.

To fix it, add a zero check to the statistics_view() endpoint like so:

# app.py

@app.route("/statistics", methods=["GET"])
def statistics_view():
    done_tasks_count = Task.query.filter_by(is_done=True).count()
    undone_tasks_count = Task.query.filter_by(is_done=False).count()

    # new
    if done_tasks_count + undone_tasks_count == 0:
        done_percentage = 0
    else:
        done_percentage = done_tasks_count / \
                          (done_tasks_count + undone_tasks_count) * 100

    return jsonify({
        "done_tasks_count": done_tasks_count,
        "undone_tasks_count": undone_tasks_count,
        "done_percentage": done_percentage,
    })

Retest the endpoint and mark it as "Closed" once you've verified the issue has been resolved.

Manual Tracking

By default, errors are only reported to AppSignal when exceptions are left unhandled. However, in some cases, you may want handled exceptions to be reported.

To accomplish this, you can utilize AppSignal's helper methods:

Wrapping Up

In this article, we've covered how to monitor errors in a Flask app using AppSignal.

We explored two error reporting methods: automatic tracking and manual tracking (using helper methods). With this knowledge, you can easily incorporate AppSignal into your Flask projects.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Track Errors in FastAPI for Python with AppSignal

Roy Tomeij — 2024-03-27T00:00:00+00:00

When you first try a new library or framework, you are excited about it. However, as soon as you run something on production, things are less than ideal — an error here, an exception there - bugs everywhere! You start reading your logs, but you often lack context, like how often an error happens, in what line, etc.

Fortunately, tools such as AppSignal can help. AppSignal helps you track your errors and gives you a lot of valuable insights. For example, you can very quickly see how often an error happens, in what line, and for which deployment.

That's what we'll do now: leverage AppSignal to track errors in a FastAPI application.

Setting Up Our FastAPI for Python Project

Let's prepare a project we'll use as an example. First, create a new directory and virtual environment:

$ mkdir fastapi_appsignal
$ cd fastapi_appsignal
$ python3.12 -m venv venv
$ source venv/bin/activate

Second, install FastAPI:

$ pip install "fastapi[all]"

Third, add a new file called main.py with the following contents:

import uuid
from uuid import UUID

from fastapi import FastAPI
from pydantic import BaseModel


class TodoData(BaseModel):
    title: str
    done: bool = False


class Todo(BaseModel):
    id: UUID
    title: str
    done: bool = False


TODOS = []
app = FastAPI()


@app.get("/todos")
def todo_list() -> list[Todo]:
    return TODOS


@app.post("/todos")
def todo_create(data: TodoData) -> Todo:
    todo = Todo(id=uuid.uuid4(), title=data.title, done=data.done)
    TODOS.append(todo)
    return todo


@app.post("/todos/{todo_id}")
def todo_edit(todo_id: UUID, data: TodoData) -> Todo:
    todo = next((t for t in TODOS if t.id == todo_id))
    todo.title = data.title
    todo.done = data.done
    return todo

What we have here is a very simple API example for managing TODOs. We'll use this API to demonstrate how to track errors with AppSignal.

Configure AppSignal with FastAPI

If you don't have an AppSignal account, create one - AppSignal offers a 30 day free trial.

First, we need to install appsignal and opentelemetry-instrumentation-fastapi:

$ pip install appsignal
$ pip install opentelemetry-instrumentation-fastapi

Second, let's configure AppSignal for FastAPI. Create a new file called __appsignal__.py with the following contents:

import os

from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="fastapi_appsignal",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
)

Third, update the main.py file:

import uuid
from uuid import UUID

from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor  # new
from pydantic import BaseModel

from __appsignal__ import appsignal  # new

appsignal.start()  # new


class TodoData(BaseModel):
    title: str
    done: bool = False


class Todo(BaseModel):
    id: UUID
    title: str
    done: bool = False


TODOS = []
app = FastAPI()


@app.get("/todos")
def todo_list() -> list[Todo]:
    return TODOS


@app.post("/todos")
def todo_create(data: TodoData) -> Todo:
    todo = Todo(id=uuid.uuid4(), title=data.title, done=data.done)
    TODOS.append(todo)
    return todo


@app.post("/todos/{todo_id}")
def todo_edit(todo_id: UUID, data: TodoData) -> Todo:
    todo = next((t for t in TODOS if t.id == todo_id))
    todo.title = data.title
    todo.done = data.done
    return todo


FastAPIInstrumentor().instrument_app(app)  # new

We need to start the AppSignal agent alongside our FastAPI application. For more details, refer to the AppSignal Python installation docs and FastAPI instrumentation docs.

At this point, our project is ready to track our first error with AppSignal. To send errors to AppSignal, you need to get your push API key. You can get one by following the official AppSignal guide for adding a new app. After you have your push API key, start the server:

$ export APPSIGNAL_PUSH_API_KEY=<your_appsignal_push_api_key>
$ uvicorn main:app --reload

Now your server is up and running, open a new terminal and send a request to the API:

$ curl -X POST -H "Content-Type: application/json" -d '{"title": "Buy milk", "done": true}' http://localhost:8000/todos/dde18df8-7e11-4f84-bcbf-2eee1a5d157e

This request will raise the StopIteration exception inside the todo_edit function. That's because we use the next function to get the TODO with the given ID. The exception is raised since we're using the UUID of a non-existing task.

To confirm that AppSignal is working, click on Next Step at the bottom of the page from which you've copied the push API key.

Once you've clicked on Next Step, wait for AppSignal to confirm that it has received data from your app. After this, you can open the Errors -> Issue list tab and see the error we've just raised.

Congrats! You've just tracked your first error in FastAPI with AppSignal. But we can do better than that. Let's see how we can get more insights into our errors.

Get More Error Insights

At this point, we're able to track errors with AppSignal. On the dashboard, you can see:

How often an error happens
Error messages
Error backtraces

You can also add notes to errors, assign errors, and post messages to Slack when errors occur.

To get even more insights, you can set up deploy markers and backtrace links.

Using them together lets you jump directly to the source code on your GitHub repository from the error's backtrace. For this to work, you need to push your code to GitHub. Do that before going further (you can follow the official guide). Once this is done, we can continue setting up deploy markers and backtrace links.

Deploy Markers

First, we need to set up deploy markers. To do that, update the __appsignal__.py file:

import os

from appsignal import Appsignal

appsignal = Appsignal(
    active=True,
    name="fastapi_appsignal",
    push_api_key=os.getenv("APPSIGNAL_PUSH_API_KEY"),
    revision=os.getenv("APPSIGNAL_REVISION"),  # new
)

We enable deploy markers by setting the revision value to a non-null value. Setting up deploy markers will allow us to see which deployment causes an error. We'll set APPSIGNAL_REVISION to main because that's the name of our branch. In the production setup, you should set this to the commit hash of the deployed version or tag name.

Second, we need to enable backtrace links. We already enabled deploy markers, so we're halfway there. What's left is to link our GitHub repository. To do that, go to your organization's settings and click on Install GitHub App.

Once on GitHub, select your organization.

After that, keep everything as it is and click Install.

You'll be redirected back to AppSignal's dashboard. Select the repository you want to link to your app and click Save repository selection.

Let's raise another error to test that everything is working as expected. To do that, set environment variables and start your server again:

$ export APPSIGNAL_PUSH_API_KEY=<your_appsignal_push_api_key>
$ export APPSIGNAL_REVISION=main
$ uvicorn main:app --reload

After that, open a new terminal and send another request to the API:

$ curl -X POST -H "Content-Type: application/json" -d '{"title": "Buy milk", "done": true}' http://localhost:8000/todos/dde18df8-7e11-4f84-bcbf-2eee1a5d157e

Backtrace Links

Once the latest error is tracked, you can try backtrace links. Go to the Errors -> Issue list tab and click on the error we've just raised. Once on the error's detail page, click Inspect latest example.

You'll end up on the details of the latest error sample, where you'll see the Backtrace section. On the right side of the backtrace, you'll see a link to the source code on GitHub - git.

When you click on git, you'll be redirected to the source code on GitHub. You'll see the exact line where the error happened. Now, you can easily pinpoint the problematic code and fix the issue. This is way less work than parsing log files and doing things manually.

Note: If you're not pointed to the correct line of code on GitHub, make sure you've committed and pushed the latest version of your code to GitHub.

Reporting Python FastAPI Errors Explicitly

So far, we've seen how to track errors raised by our FastAPI application. But we might want to report errors explicitly in some cases. For example, we might use an AI API to generate emails for our clients.

As you may know, these APIs are often unreliable, and seeing internal server errors is quite common. In such cases, we want to handle the error - for example, by falling back to a default template.

We want to report an error to AppSignal to see how often it happens and what the error message is.

Using `set_error`

In some cases, we can make usage more robust (e.g., by implementing retries) or contact the API provider's support. To handle such scenarios, AppSignal provides the set_error function. Let's see how to use it.

Update your main.py file:

# ... existing imports
from appsignal import set_error  # new
from fastapi import FastAPI, HTTPException  # new
# ... existing imports


# ... existing code

@app.post("/todos/{todo_id}")
def todo_edit(todo_id: UUID, data: TodoData) -> Todo:
    try:  # new
        todo = next((t for t in TODOS if t.id == todo_id))
    except StopIteration:
        set_error(Exception("Todo not found"))
        raise HTTPException(status_code=404, detail="Todo not found")
    todo.title = data.title
    todo.done = data.done
    return todo


# ... existing code

In our case, users won't see the Internal server error anymore. Instead, we'll return the Todo not found error. That's better for our users and our API. Without set_error, nothing would be reported to AppSignal. With set_error, the error is reported and tracked as before. Hence, we'll see how often this error happens and the error message. Backtrace links still apply, as in the previous example.

To test that everything is working as expected, follow the same steps as above:

Start the server.
Send a request to the API.
Go to the dashboard Errors -> Issue list in AppSignal and check the new error.

That's all great, but sometimes, we need more context when manually reporting an error to AppSignal. In such cases, we can use the send_error_with_context function. This function allows us to manually add needed context to the error - e.g., the ID of a non-existing task. Once again, update the main.py file:

# ... existing imports
from appsignal import send_error_with_context, set_params  # new
from fastapi import FastAPI, HTTPException  # new
# ... existing imports


# ... existing code

@app.post("/todos/{todo_id}")
def todo_edit(todo_id: UUID, data: TodoData) -> Todo:
    try:
        todo = next((t for t in TODOS if t.id == todo_id))
    except StopIteration:
        with send_error_with_context(Exception("Todo not found")):
            set_params({"todo_id": todo_id})
        raise HTTPException(status_code=404, detail="Todo not found")
    todo.title = data.title
    todo.done = data.done
    return todo



# ... existing code

To test that everything works as expected, follow the same steps as above.

In this case, you'll also see the Parameters section that contains the parameters you've explicitly set (here, that's the ID of the non-existing task).

And that's it!

Wrapping Up

In this blog post, we've seen how to track errors in FastAPI with AppSignal. We set up AppSignal for FastAPI, tracked errors, and gained some further insights.

Besides that, we also learned how to explicitly track errors using the AppSignal functions set_error and send_error_with_context. This should give you a good starting point to track your errors better and fix them faster from now on.

Happy engineering!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Track Errors in Your Python Django Application with AppSignal

Roy Tomeij — 2024-02-28T00:00:00+00:00

In this post, we will specifically look at using AppSignal to track errors in a Django application.

We'll first create a Django project, install AppSignal, introduce some faulty code, and then use the AppSignal Errors dashboard to debug and resolve errors.

Let's get started!

Prerequisites

To follow along, you'll need:

Python 3.8+ installed on your local machine
An AppSignal-supported operating system
An AppSignal account
Basic Django knowledge

Project Setup

We'll be working on a movie review web app. The app will allow us to manage movies and reviews via a RESTful API. To build it, we'll utilize Django.

I recommend you follow along with the movie review web app first. After you grasp the basic concepts, using AppSignal with your projects will be easy.

Begin by creating and activating a new virtual environment, installing Django, and bootstrapping a new Django project.

If you need help, refer to the Quick install guide from the Django docs.

Note: The source code for this project can be found on the appsignal-django-error-tracking GitHub repo.

Install AppSignal for Django

To add AppSignal to a Django-based project, follow the AppSignal documentation:

To ensure everything works, start the development server. Your Django project will automatically send a demo error to AppSignal. In addition, you should receive an email notification.

From now on, all your app errors will be reported to AppSignal.

App Logic

Moving along, let's implement the movie review app logic.

You might notice that some code snippets contain faulty code. The defective code is placed there on purpose to later demonstrate how AppSignal error tracking works.

First, create a dedicated app for movies and reviews:

(venv)$ python manage.py startapp movies

Add the newly-created app to INSTALLED_APPS in settings.py:

# core/settings.py

INSTALLED_APPS = [
    # ...
    'movies.apps.MoviesConfig',
]

Define the app's database models in models.py:

# movies/models.py

from django.db import models


class Movie(models.Model):
    title = models.CharField(max_length=128)
    description = models.TextField(max_length=512)
    release_year = models.IntegerField()

    def get_average_rating(self):
        reviews = MovieReview.objects.filter(movie=self)
        return sum(review.rating for review in reviews) / len(reviews)

    def serialize_to_json(self):
        return {
            'id': self.id,
            'title': self.title,
            'description': self.description,
            'release_year': self.release_year,
        }

    def __str__(self):
        return f'{self.title} ({self.release_year})'


class MovieReview(models.Model):
    movie = models.ForeignKey(Movie, on_delete=models.CASCADE)
    rating = models.IntegerField()

    def save(self, force_insert=False, force_update=False, using=None, update=None):
        if self.rating < 1 or self.rating > 5:
            raise ValueError('Rating must be between 1 and 5.')

        super().save(force_insert, force_update, using, update)

    def serialize_to_json(self):
        return {
            'id': self.id,
            'movie': self.movie.serialize_to_json(),
            'rating': self.rating,
        }

    def __str__(self):
        return f'{self.movie.title} - {self.rating}'

What's Happening Here?

We define two models, Movie and MovieReview. A Movie can have multiple MovieReviews. Both models include a serialize_to_json() method for serializing the object to JSON.
The Movie model includes get_average_rating() for calculating a movie's average rating.
MovieReview's rating is locked in a [1, 5] interval via the modified save() method.

Now, make migrations and migrate the database:

(venv)$ python manage.py makemigrations
(venv)$ python manage.py migrate

Define the Views in Python

Next, define the views in movies/views.py:

# movies/views.py

from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_http_methods

from movies.models import Movie, MovieReview


def index_view(request):
    queryset = Movie.objects.all()
    data = [movie.serialize_to_json() for movie in queryset]

    return JsonResponse(data, safe=False)


def statistics_view(request):
    queryset = Movie.objects.all()
    data = []

    for movie in queryset:
        data.append({
            'id': movie.id,
            'title': movie.title,
            'average_rating': movie.get_average_rating(),
        })

    return JsonResponse(data, safe=False)


@csrf_exempt
@require_http_methods(["POST"])
def review_view(request):
    movie_id = request.POST.get('movie')
    rating = request.POST.get('rating')

    if (movie_id is None or rating is None) or \
            (not movie_id.isdigit() or not rating.isdigit()):
        return JsonResponse({
            'detail': 'Please provide a `movie` (int) and `rating` (int).',
        }, status=400)

    movie_id = int(movie_id)
    rating = int(rating)

    try:
        movie = Movie.objects.get(id=movie_id)
        MovieReview.objects.create(
            movie=movie,
            rating=rating,
        )

        return JsonResponse({
            'detail': 'A review has been successfully posted',
        })

    except Movie.DoesNotExist:
        return JsonResponse({
            'detail': 'Movie does not exist.',
        }, status=400)

What's Happening Here?

We define three views: index_view(), statistics_view(), and review_view().
The index_view() fetches all the movies, serializes them, and returns them.
The statistics_view() calculates and returns the average rating of every movie.
The review_view() allows users to create a movie review by providing the movie ID and a rating.

Register the URLs

The last thing we must do is take care of the URLs.

Create a urls.py file within the movies app with the following content:

# movies/urls.py

from django.urls import path

from movies import views

urlpatterns = [
    path('statistics/', views.statistics_view, name='movies-statistics'),
    path('review/', views.review_view, name='movies-review'),
    path('', views.index_view, name='movies-index'),
]

Then register the app URLs globally:

# core/urls.py

from django.contrib import admin
from django.urls import path, include  # new import

urlpatterns = [
    path('movies/', include('movies.urls')),  # new
    path('admin/', admin.site.urls),
]

Using Fixtures

To get some test data to work with, I've prepared two fixtures.

First, download the fixtures, create a fixtures folder in the project root, and place them in there:

django-error-tracking/
+-- fixtures/
    +-- Movie.json
    +-- MovieReview.json

After that, run the following two commands to load them:

(venv)$ python manage.py loaddata fixtures/Movie.json --app app.Movie
(venv)$ python manage.py loaddata fixtures/MovieReview.json --app app.MovieReview

We now have a functional API with some sample data to work with. In the next section, we'll test it.

Test Your Django App's Errors with AppSignal

During the development of our web app, we left intentional bugs in the code. We will now deliberately trigger these bugs to see what happens when an error occurs.

Before proceeding, ensure your Django development server is running:

(venv)$ python manage.py runserver

Your API should be accessible at http://localhost:8000/movies.

AppSignal should, of course, be employed when your application is in production rather than during development, as shown in this article.

Error 1: `ZeroDivisionError`

Start by visiting http://localhost:8000/movies/statistics in your favorite browser. This endpoint is supposed to calculate and return average movie ratings, but it results in a ZeroDivisonError.

Let's use AppSignal to figure out what went wrong.

First, navigate to your AppSignal dashboard and select your application. On the sidebar, you'll see several categories. Select "Errors > Issue list":

You'll see that a ZeroDivisionError has been reported. Click on it to inspect it.

The error detail page displays the error's summary, trends, state, severity, etc. But we are interested in the samples. Select the "Samples" menu item in the navigation bar to access them.

A sample refers to a recorded instance of a specific error. Select the first sample.

The sample's detail page shows the error message, what device triggered the error, the backtrace, and more. We can look at the backtrace to determine what line of code caused the error.

In the backtrace, we can see that the error occurred in movies/models.py on line 16. Looking at the code, the error is obvious: if a movie has no reviews, this results in a division by zero.

To fix this error, all you have to do is slightly modify the Movie's get_average_rating() method:

# movies/models.py

class Movie(models.Model):
    # ...

    def get_average_rating(self):
        reviews = MovieReview.objects.filter(movie=self)

        # new condition
        if len(reviews) == 0:
            return 0

        return sum(review.rating for review in reviews) / len(reviews)

    # ...

Reload the development server, test the functionality again, and set the error's state to "Closed" if everything works as expected.

Error 2: `ValueError`

We can trigger another error by submitting a review with a rating outside the [1, 5] interval. To try it out, open your terminal and run the following command:

$ curl --location --request POST 'http://localhost:8000/movies/review/' \
     --form 'movie=2' \
     --form 'rating=6'

As expected, a ValueError is reported to AppSignal. To track the error, follow the same procedure as in the previous section.

The error can be easily fixed by adding a check to review_view() in views.py:

# movies/views.py

from django.core.validators import MinValueValidator, MaxValueValidator

# ...

@csrf_exempt
@require_http_methods(["POST"])
def review_view(request):
    # ...

    movie_id = int(movie_id)
    rating = int(rating)

    if rating < 1 or rating > 5:
        return JsonResponse({
            'detail': 'Rating must be between 1 and 5.',
        }, status=400)

    try:
        movie = Movie.objects.get(id=movie_id)
        MovieReview.objects.create(
            movie=movie,
            rating=rating,
        )
        # ...

    # ...

After that, test the functionality again and tag the error as "Resolved" if everything works as expected.

Wrapping Up

In this article, you've learned how to use AppSignal to track errors in a Django application.

To get the most out of AppSignal for Python, I suggest you review the following two resources:

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

An Introduction to Testing with Django for Python

Roy Tomeij — 2024-01-31T00:00:00+00:00

In a world of ever-changing technology, testing is an integral part of writing robust and reliable software. Tests verify that your code behaves as expected, make it easier to maintain and refactor code, and serve as documentation for your code.

There are two widely used testing frameworks for testing Django applications:

Django's built-in test framework, built on Python's unittest
Pytest, combined with pytest-django

In this article, we will see how both work.

Let's get started!

What Will We Test?

All the tests we'll observe will use the same code, a BookList endpoint.

The Book model has title, author, and date_published fields. Note that the default ordering is set by date_published.

# models.py
class Book(models.Model):
    title = models.CharField(max_length=20)
    author = models.ForeignKey(Author, on_delete=models.CASCADE, blank=True, null=True)
    date_published = models.DateField(null=True, blank=True)

    class Meta:
        ordering = ['date_published']

The view is just a simple ListView:

# views.py
from django.views.generic import ListView
from .models import Book


class BookListView(ListView):
    model = Book

Since the test examples will test the endpoint, we also need the URL:

# urls.py
from django.urls import path
from . import views

urlpatterns = [
    path("books/", views.BookListView.as_view(), name="book_list"),
]

Now for the tests.

Unittest for Python

Unittest is Python's built-in testing framework. Django extends it with some of its own functionality.

Initially, you might be confused about what methods belong to unittest and what are Django's extensions.

Unittest provides:

TestCase, serving as a base class.
setUp() and tearDown() methods for the code you want to execute before or after each test method. setUpClass() and tearDownClass() also run once per whole TestClass.
Multiple types of assertions, such as assertEqual, assertAlmostEqual, and assertIsInstance.

The core part of the Django testing framework is TestCase.

Since it subclasses unittest.TestCase, TestCase has all the same functionality but also builds on top of it, with:

setUpTestData class method: This creates data once per TestCase (unlike setUp, which creates test data once per test). Using setUpTestData can significantly speed up your tests.
Test data loaded via fixtures.
Django-specific assertions, such as assertQuerySetEqual, assertFormError, and assertContains.
Temporary override settings you can set up during a test run.

Django's testing framework also provides a Client that doesn't rely on TestCase and so can be used with pytest. Client serves as a dummy browser, allowing users to create GET and POST requests.

How Tests Are Built with Unittest

Unittest supports test discovery, but the following rules must be followed:

The test should be in a file named with a test prefix.
The test must be within a class that subclasses unittest.TestCase (that includes using django.TestCase). Including 'Test' in the class name is not mandatory.
The name of the test method must start with test_.

Let's see what tests written with unittest look like:

# tests.py
from datetime import date

from django.test import TestCase
from django.urls import reverse
from .models import Book, Author

class BookListTest(TestCase):
    @classmethod
    def setUpTestData(cls):
        author = Author.objects.create(first_name='Jane', last_name='Austen')
        cls.second_book = Book.objects.create(title='Emma', author=author, date_published=date(1815, 1, 1))
        cls.first_book = Book.objects.create(title='Pride and Prejudice', author=author, date_published=date(1813, 1, 1))

    def test_book_list_returns_all_books(self):
        response = self.client.get(reverse('book_list'))
        response_book_list = response.context['book_list']

        self.assertEqual(response.status_code, 200)
        self.assertIn(self.first_book, response_book_list)
        self.assertIn(self.second_book, response_book_list)
        self.assertEqual(len(response_book_list), 2)

    def test_book_list_ordered_by_date_published(self):
        response = self.client.get(reverse('book_list'))
        books = list(response.context['book_list'])

        self.assertEqual(response.status_code, 200)
        self.assertQuerySetEqual(books, [self.first_book, self.second_book])

You must always put your test functions inside a class that extends a TestCase class. Although it's not required, we've included 'Test' in our class name so it's easily recognizable as a test class at first glance. Notice that the TestCase is imported from Django, not unittest — you want access to additional Django functionality.

What's Happening Here?

This TestCase aims to check whether the book_list endpoint works correctly. There are two tests — one checks that all Book objects in the database are included in the response as book_list, and the second checks that the books are listed by ascending publishing date.

Although the tests appear similar, they evaluate two distinct functionalities, which should not be combined into a single test.

Since both tests need the same data in the database, we use the class method setUpTestData for preparing data — both tests will have access to it without repeating the process twice. Unlike the two Book objects, the Author object is not needed outside the setup method, so we don't set it as an instance attribute. I switched the order of the two books to ensure that the correct order isn't accidental.

The two tests look very similar — they both make a GET request to the same URL and extract the book_list from context. Methods inside the TestCase class automatically have access to client, which is used to make a request.

Up to this point, both tests did the same thing — but the assertions differ.

The first test asserts that both books are in the response context, using assertIn. It also ensures the length of the response.context['book_list'] object. The second test uses Django's assertQuerySetEqual. This assertion checks the ordering by default, so it does exactly what we need.

Both tests also check the status_code, so you know immediately if the URL doesn't work.

Running Our Test with Unittest

We can run a test with Django's test command:

(venv)$ python manage.py test

This is the output:

Found 2 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.089s

OK
Destroying test database for alias 'default'...

What if we run a failing test? For example, if the order of returned objects doesn't match the desired result:

(venv)$ python manage.py test
Found 2 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
F
======================================================================
FAIL: test_book_order (bookstore.test.BookListTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bookstore/test.py", line 17, in test_book_order
    self.assertEqual(books, [self.second_book, self.first_book])
AssertionError: Lists differ: [<Book: Pride and Prejudice>, <Book: Emma>] != [<Book: Emma>, <Book: Pride and Prejudice>]

First differing element 0:
<Book: Pride and Prejudice>
<Book: Emma>

- [<Book: Pride and Prejudice>, <Book: Emma>]
+ [<Book: Emma>, <Book: Pride and Prejudice>]
----------------------------------------------------------------------
Ran 2 tests in 0.085s

FAILED (failures=1)
Destroying test database for alias 'default'...

As you can see, the output is quite informative — you learn which test failed, in which line, and why. The test failed because the lists differ; you can even see both lists, so you can quickly determine where the problem lies.

Another essential thing to notice here is that only the second test failed — figuring out what's wrong can be more challenging if there's a single test.

Pytest for Python

Pytest is an excellent alternative to unittest. Even though it doesn't come built-in to Python itself, it is considered more pythonic than unittest. It doesn't require a TestClass, has less boilerplate code, and has a plain assert statement. Pytest has a rich plugin ecosystem, including a specific Django plugin, pytest-django.

The pytest approach is quite different from unittest.

Among other things:

You can write function-based tests without the need for classes.
It allows you to create fixtures: reusable components for setup and teardown. Fixtures can have different scopes (e.g., module), allowing you to optimize performance while keeping a test isolated.
Parametrization of the test function means it can be efficiently run with different arguments.
It has a single, plain assert statement.

Pytest and Django

pytest-django serves as an adapter between Pytest and Django. Testing Django with pytest without pytest-django is technically possible, but not practical nor recommended. Along with handling Django's settings, static files, and templates, it brings some Django test tools to pytest, but also adds its own:

A @pytest.mark.django_db decorator that enables database access for a specific test.
A client that passes Django's Client in the form of a fixture.
The admin_client fixture returns an authenticated Client with admin access.
A settings fixture that allows you to temporarily override Django settings during a test.
The same special Django assertions as Django TestCase.

How Tests Are Built with Pytest

Like unittest, pytest also supports test discovery. As you'll see below, the out-of-the-box rules are (these rules can be easily changed):

The files must be named test_*.py or *_test.py.
There's no need for test classes, but if you want to use them, the name should have a Test prefix.
The function names need to be prefixed with test.

Let's see what tests written with pytest look like:

from datetime import date

import pytest
from django.urls import reverse

from bookstore.models import Author, Book


@pytest.mark.django_db
def test_book_list_returns_all_books(client):
    author = Author.objects.create(first_name='Jane', last_name='Austen')
    second_book = Book.objects.create(title='Emma', author=author, date_published=date(1815, 1, 1))
    first_book = Book.objects.create(title='Pride and Prejudice', author=author, date_published=date(1813, 1, 1))

    response = client.get(reverse('book_list'))
    books = list(response.context['book_list'])

    assert response.status_code == 200
    assert first_book in books
    assert second_book in books
    assert len(books) == 2


@pytest.mark.django_db
def test_book_list_ordered_by_date_published(client):
    author = Author.objects.create(first_name='Jane', last_name='Austen')
    second_book = Book.objects.create(title='Emma', author=author, date_published=date(1815, 1, 1))
    first_book = Book.objects.create(title='Pride and Prejudice', author=author, date_published=date(1813, 1, 1))

    response = client.get(reverse('book_list'))
    books = list(response.context['book_list'])

    assert response.status_code == 200
    assert books == [first_book, second_book]

What's Happening Here?

These two tests assess the same functionality as the previous two unittest tests — the first test ensures all the books in the database are returned, and the second ensures the books are ordered by publication date. How do they differ?

They're entirely standalone — no class or joint data preparation is needed. You could put them in two different files, and nothing would change.
Since they need database access, they require a @pytest.mark.django_db decorator that comes from pytest-django. If there's no DB access required, skip the decorator.
Django's client needs to be passed as a fixture since you don't automatically have access to it. Again, this comes from pytest-django.
Instead of different assert statements, a plain assert is used.

The lines that execute the request and that turn the book_list in the response are precisely the same as in unittest.

Fixtures in Pytest

In pytest, there's no setUpTestData. But if you find yourself repeating the same data preparation code over and over again, you can use a fixture.

The fixture in our example would look like this:

# fixture creation
import pytest

@pytest.fixture
def two_book_objects_same_author():
    author = Author.objects.create(first_name='Jane', last_name='Austen')
    second_book = Book.objects.create(title='Emma', author=author, date_published=date(1815, 1, 1))
    first_book = Book.objects.create(title='Pride and Prejudice', author=author, date_published=date(1813, 1, 1))

    return [first_book, second_book]

# fixture usage
@pytest.mark.django_db
def test_book_list_ordered_by_date_published_with_fixture(client, two_book_objects_same_author):
    response = client.get(reverse('book_list'))
    books = list(response.context['book_list'])

    assert response.status_code == 200
    assert books == two_book_objects_same_author

You mark a fixture with the @pytest.fixture decorator. You then need to pass it as an argument for a function to use. You can use fixtures to avoid repetitive code in the preparation step or to improve readability, but don't overdo it.

Running a Test with Pytest for Django

For pytest to work correctly, you need to tell django-pytest where Django's settings can be found. While this can be done in the terminal (pytest --ds=bookstore.settings), a better option is to use pytest.ini.

At the same time, you can use pytest.ini to provide other configuration options.

pytest.ini looks something like this:

[pytest]

;where the django settings are
DJANGO_SETTINGS_MODULE = bookstore.settings

;changing test discovery
python_files = tests.py test_*.py

;output logging records into the console
log_cli = True

Once DJANGO_SETTINGS_MODULE is set, you can simply run the tests with:

(venv)$ pytest

Let's see the output:

================================================================================================= test session starts ==================================================================================================
platform darwin -- Python 3.10.5, pytest-7.4.3, pluggy-1.3.0
django: settings: bookstore.settings (from option)
rootdir: /bookstore
plugins: django-4.5.2
collected 2 items

bookstore/test_with_pytest.py::test_book_list_returns_all_books_with_pytest PASSED                                                                                                                                   [ 50%]
bookstore/test_with_pytest.py::test_book_list_ordered_by_date_published_with_pytest PASSED                                                                                                                           [100%]

================================================================================================== 2 passed in 0.58s ===================================================================================================

This output is shown if log_cli is set to True. If it's not explicitly set, it defaults to False, and the output of each test is replaced by a simple dot.

And what does the output look like if the test fails?

================================================================================================= test session starts ==================================================================================================
platform darwin -- Python 3.10.5, pytest-7.4.3, pluggy-1.3.0
django: settings: bookstore.settings (from ini)
rootdir: /bookstore
configfile: pytest.ini
plugins: django-4.5.2
collected 2 items

bookstore/test_with_pytest.py::test_book_list_returns_all_books_with_pytest PASSED                                                                                                                                   [ 50%]
bookstore/test_with_pytest.py::test_book_list_ordered_by_date_published_with_pytest FAILED                                                                                                                           [100%]

======================================================================================================= FAILURES =======================================================================================================
___________________________________________________________________________________________________ test_book_order ____________________________________________________________________________________________________

client = <django.test.client.Client object at 0x1064dbfa0>, books_in_correct_order = [<Book: Emma>, <Book: Pride and Prejudice>]

    @pytest.mark.django_db
    def test_book_order(client, books_in_correct_order):
        response = client.get(reverse('book_list'))
        books = list(response.context['book_list'])

>       assert books == books_in_correct_order
E       assert [<Book: Pride... <Book: Emma>] == [<Book: Emma>...nd Prejudice>]
E         At index 0 diff: <Book: Pride and Prejudice> != <Book: Emma>
E         Full diff:
E         - [<Book: Emma>, <Book: Pride and Prejudice>]
E         + [<Book: Pride and Prejudice>, <Book: Emma>]

=============================================================================================== short test summary info ================================================================================================
FAILED bookstore/test_with_pytest.py::test_book_list_ordered_by_date_published_with_pytest - assert [<Book: Pride... <Book: Emma>] == [<Book: Emma>...nd Prejudice>]
============================================================================================= 1 failed, 1 passed in 0.78s ==============================================================================================

If log_cli is set to True, you can easily see which test passed and which failed. Although there's a long output on the error, you can often find enough information on why the test failed in the short test summary info.

Unittest vs. Pytest for Python

There's no consensus as to whether unittest or pytest is better. The opinions of developers differ, as you can see on StackOverflow or Reddit. A lot depends on your own preferences or those of your team. If most of your team is used to pytest, there is no need to switch, and vice-versa.

However, if you don't have your preferred way of testing yet, here's a quick comparison between the two:

unittest (with Django test)	pytest (with pytest-django)
No additional installation needed	Installation of pytest and pytest-django required
Class-based approach	Functional approach
Multiple types of assertions	A plain `assert` statement
Setup/TearDown methods within `TestCase` class	Fixture system with different scopes
Parametrization isn't supported (but it can be done on your own)	Native support for parametrization

How to Approach Tests in Django

Django encourages rapid development, so you might feel that you don't actually need tests as they would just slow you down. However, tests will, in the long run, make your development process faster and less error-prone. Since Django provides a lot of building blocks that allow you to accomplish much with only a few lines of code, you may end up writing way more tests than actual code.

One thing that can help you cover most of your code with tests is code coverage. "Code coverage" is a measure that tells you what percentage of your program's code is executed during a test suite run. The higher the percentage, the bigger the portion of the code being tested, which usually means fewer unnoticed problems.

Coverage.py is the go-to tool for measuring code coverage of Python programs. Once installed, you can use it with either unittest or pytest.

If using pytest, you can install pytest-cov — a library that integrates Coverage.py with pytest. It is widely used, but the coverage.py official documentation states that it's mostly unnecessary.

The commands for running coverage are a little different for unittest and pytest:

For unittest: coverage run manage.py test
For pytest: coverage run -m pytest

Those commands will run tests and check the coverage, but you must run a separate command to get the output.

To see the report in your terminal, you need to run the coverage report command:

$(venv) coverage report
Name                                                                             Stmts   Miss  Cover
------------------------------------------------------------------------------------------------------
core/__init__.py                                                         0      0    100%
core/settings.py                                                         34     0    100%
core/urls.py                                                             3      0    100%
bookstore/__init__.py                                                             0      0    100%
bookstore/admin.py                                                                19     1    95%
bookstore/apps.py                                                                 6      0    100%
bookstore/models.py                                                               64     7    89%
bookstore/urls.py                                                                 3      0    100%
bookstore/views.py                                                                34     7    79%
------------------------------------------------------------------------------------------------------
TOTAL                                                                             221     15    93%

Percentages ranging from 75% to 100% are seen as giving "good coverage". Aim for a score higher than 75%, but keep in mind that just because your code is well-covered by tests, doesn't mean it is any good.

A Note on Tests

Test coverage is one aspect of a good test suite. However, as long as your coverage is not below 75%, the exact percentage is not that significant.

Don't write tests just for the sake of writing them or to improve your coverage percentage. Not everything in Django needs testing — writing useless tests will slow down your test suite and make refactoring a slow and painful process.

You should not test Django's own code — it's already been tested. For example, you don't need to write a test that checks if an object is retrieved with get_object_or_404 — Django's testing suite already has that covered.

Also, not every implementation detail needs to be tested. For example, I rarely check if the correct template is used for a response — if I change the name, the test will fail without providing any real value.

How Can AppSignal Help with Testing in Django?

The trouble with tests is that they're written in isolation — when your app is live, it might not behave the same, and the user will probably not behave just as you expected.

That's why it's a good idea to use application monitoring — it can help you track errors and monitor performance. Most importantly, it can inform you when an error is fatal, breaking your app.

AppSignal integrates with Django and can help you find the errors you miss when writing tests.

With AppSignal, you can see how often an error occurs and when. Knowing that can help you decide how urgently you need to fix the error.

Since you can integrate AppSignal with your Git repository, you can also pinpoint the line in which an error occurs. This simplifies the process of identifying an error's cause, making it easier to fix. When an error is well understood, it's also easier to add tests that prevent it and similar errors in the future.

Here's an example of an error from the Issues list under the Errors dashboard for a Django application in AppSignal:

Wrapping Up

In this post, we've seen that you have two excellent options for testing in Django: unittest and pytest. We introduced both options, highlighting their unique strengths and capabilities. Which one you choose largely hinges on your personal or project-specific preferences.

I hope you've seen how writing tests is essential to having a high-quality product that provides a seamless user experience.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

How to Deploy a Python Flask app with Heroku

Roy Tomeij — 2023-12-06T00:00:00+00:00

In this tutorial, we will build a simple Flask app that is primed and ready to deploy to Heroku.

Once the bare bones of the app are built, we will guide you through the setup process on GitHub and Heroku so that you can start making automatic deploys in no time.

But before we dive straight into the code, why choose Flask and Heroku in the first place?

Why Flask for Python?

Flask is a super-minimalistic web framework for Python. It provides only the absolute core functionality needed to build web applications in a small and nimble package.

One major advantage of this approach is that Flask provides web developers with maximum flexibility to design and build their app from the very beginning.

On the other hand, frameworks such as Django prefer the "batteries-included" philosophy. This means more is taken care of for you out-of-the-box, but there will be more boilerplate code. The structure of your application is more rigid and harder to change later on.

Also, as the industry moves away from big code "monoliths" built on frameworks like Django, and towards smaller microservice architecture, it has never been a better time to get familiar with Flask.

Why Heroku?

When Heroku launched in 2007, it was one of the first cloud platforms renowned for its user-friendly interface and ease of use. Using Heroku was a huge time-saver, as it made deploying web applications much easier than configuring all your infrastructure from scratch using AWS. In fact, Heroku itself is built on AWS infrastructure. Another advantage was the fact that you could deploy your hobby app with a full Postgres database totally for free.

Since then, several other cloud services have sprung up to challenge Heroku's first-mover advantage in the sector of cloud computing known as Platform-as-a-Service (PaaS).

But Heroku still retains one significant edge over the newer cloud providers: its extensive library of third-party add-ons.

If you need functionality for your app like error tracking or performance monitoring (which, incidentally, AppSignal provides) you can simply install an add-on for that on Heroku's interface and get started immediately. For many developers, this speed and convenience might make the difference between shipping or not shipping.

Create a Python Flask Demo App for Heroku Deployment

The very first thing we need to do is create an empty Flask project. Lets call it flask-heroku-appsignal. We'll also create the main entry point file for our project, app.py:

mkdir flask_heroku_appsignal
cd flask-heroku-appsignal
touch app.py

Then we'll make our project a git repository by running the git initialize command in the root of the directory:

git init

By the way, if you want to view the full code for this article at any point, download or clone it here.

Next, we need to install the core dependencies, which include Flask itself, python-dotenv (to manage environment variables), and gunicorn, which will act as the "bridge" between Heroku's web servers and our web application.

Create a requirements.txt file in the root of the project and just add the dependency names to it (you don't need to include the version numbers):

Flask
python-dotenv
gunicorn

Then run:

pip install -r requirements.txt

If everything goes smoothly, we should see some output that looks something like this:

Collecting Flask
  Using cached Flask-2.2.3-py3-none-any.whl (101 kB)
Collecting python-dotenv
  Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)
Collecting Werkzeug>=2.2.2
  Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
Collecting itsdangerous>=2.0
  Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting click>=8.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting Jinja2>=3.0
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.2-cp310-cp310-macosx_10_9_x86_64.whl (13 kB)
Installing collected packages: python-dotenv, MarkupSafe, itsdangerous, click, Werkzeug, Jinja2, Flask
Successfully installed Flask-2.2.3 Jinja2-3.1.2 MarkupSafe-2.1.2 Werkzeug-2.2.3 click-8.1.3 itsdangerous-2.1.2 python-dotenv-1.0.0

Preparing for Heroku Deployment

Now that all our basic project foundations are in place, let’s start building and configuring our Flask app with deployment to Heroku in mind.

First, let's copy this super-simple "hello world" code into app.py to check everything is working as expected so far:

from flask import Flask
from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)

@app.route("/")
def index():
  return "Hello World!"

Then type flask run in the terminal. We should see the following output:

* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

When we go to localhost:5000, we will see the Hello World! string printed in the browser.

At this point, I have a nice time-saving Flask tip to share with you that isn't mentioned in most Flask tutorials.

By adding the simple one-liner below in your .env at the root of the project, you can save lots of time. You won't have to stop and start your local server every time you want to make changes to your backend code. While this doesn't sound like a huge thing, these small changes add up to loads of compounded saved time over the long run.

FLASK_DEBUG=True

As the “debug” name suggests, this setting will provide a nicely formatted error read-out in the browser to make it easier to debug your code.

Also, ensure that the .env file is included in your project's .gitignore so that it's not tracked by git. This means it will not be picked up by GitHub and Heroku (as we don’t want the debug mode to be part of our production settings).

After making this change, the output on our terminal will display Debug mode as on with the Debugger PIN:

* Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 133-697-373

Development and Production Settings In Our Python Flask App

Next, we want to make a clear distinction between the development and production settings in our Flask app. This is an important step in preparing for deployment to Heroku. To do this, we will create a config.py file in the root of the project and add the following code with three classes:

class Config:
 DEBUG = False
 DEVELOPMENT = False
 CSRF_ENABLED = True
 ASSETS_DEBUG = False

class ProductionConfig(Config):
 pass

class DevelopmentConfig(Config):
 DEBUG = True
 DEVELOPMENT = True
 TEMPLATES_AUTO_RELOAD = True
 ASSETS_DEBUG = True

Back in app.py, add the following two lines before the index route function:

...
env_config = os.getenv("PROD_APP_SETTINGS", "config.DevelopmentConfig")
app.config.from_object(env_config)

So now the full app.py will look like this:

import os
from flask import Flask
from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)

env_config = os.getenv("PROD_APP_SETTINGS", "config.DevelopmentConfig")
app.config.from_object(env_config)

@app.route("/")
def index():
    return "Hello World!"

Essentially, the code above says that if we find a key called PROD_APP_SETTINGS in our environment, the config for our app should be set to config.ProductionConfig. Otherwise, we can safely assume we are in development mode and set it to config.DevelopmentConfig.

Now we are ready to set up our app in Heroku's web interface.

GitHub Setup

Before setting up on Heroku, we first need to create a repository on GitHub.

Go to github.com and create a new account (if you don't have one already).

We can call our GitHub repository the same name as our local Flask project:

Now we simply need to follow GitHub's instructions and push the code from our existing local repository to the remote one.

To double-check that the local and remote repositories have been linked correctly, run this command in the project directory:

git remote -v

We should get something very similar to the following output:

origin  https://github.com/daneasterman/flask-heroku-appsignal.git (fetch)
origin  https://github.com/daneasterman/flask-heroku-appsignal.git (push)

With the GitHub steps complete, we can start getting set up on Heroku by first creating a new app.

Heroku Deployment

We need to make sure that we have Heroku's special Procfile at the root of our project. The Procfile is a configuration text file specifying the various process types that will run when your app starts up. Without this, your app will not run on Heroku.

Copy the line below and add it to your Procfile:

web: gunicorn app:app

Also, remember the config.py we created in the previous section? We need to update our configuration in Heroku so that the app runs on our production settings, with DEBUG set to False.

First, click on the Setting tab on your Heroku dashboard. Then click the Reveal Config Vars button. Add the variable PROD_APP_SETTINGS and the value config.ProductionConfig.

Next, we need to connect our GitHub repository to Heroku. Click on the "Connect to GitHub" button. Then in the section that appears underneath, search for our flask-heroku-appsignal repository, and finally click on the second "Connect" button below.

Then click the "Enable Automatic Deploys" button. This means that every time you push your code to the main branch on GitHub, it will trigger an automatic deployment to Heroku.

Unfortunately, Heroku doesn't provide a free tier for demo apps anymore. So to complete the proof-of-concept for our flask-heroku-appsignal app, we need to purchase a dyno-type subscription.

This is where Heroku is a little cheeky and automatically puts you in the slightly more expensive Basic Plan. To change this, click on the Resources tab and then click: Change Dyno Type, which will bring up a modal screen. Finally select the Eco dyno, which will be absolutely fine for our purposes:

And we're done!

Wrapping Up

That concludes this introduction to deploying your Flask app to Heroku. Once you build out your web app with more functionality, you can think about installing the AppSignal add-on for Heroku, which now supports Python and Flask apps.

With this add-on, you can track errors, monitor performance, get access to advanced metric dashboards, and more. Happy building!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

AppSignal Monitoring Available for Python Applications

Roy Tomeij — 2023-10-31T00:00:00+00:00

We're happy to announce that AppSignal now offers monitoring tools for Python projects.

AppSignal helps you get the most out of your Python application's monitoring metrics, with additional support for multiple Python frameworks and packages such as Django and Celery.

In this article, we'll walk you through some of our core features to show you how to power up your Python application with AppSignal.

Monitoring Support for Popular Python Libraries

AppSignal already supports various popular Python libraries, which will only expand with time! We currently offer support for:

Celery
Django
FastAPI
Flask
Jinja2
Psycopg2
Redis
Requests
Starlette
WSGI/ASGI

Is there a Python package we don't support? Reach out to us, and we'll investigate it further with you.

Error Monitoring

AppSignal is a tool that tells you when, why, and where errors occur in your application.

AppSignal can be configured to track errors per deployment, and tools like Time Detective give you a helicopter view of your application. When an error occurs, you can connect the dots and spend more time solving problems rather than speculating over them.

AppSignal gives you control of when and where you're notified about errors in your application, whether that's on Slack or Discord; you can even connect AppSignal to popular collaboration tools like GitHub and Jira and leave crucial contextual information about an error in the Logbook.

Performance Monitoring

AppSignal monitors your application and tells you when its performance is suboptimal. Spot and fix performance bottlenecks so you can scale your app with confidence.

Our intuitive UI helps you see where and when performance problems arise, with all the critical insights you need at your fingertips to choose the best resolution.

Anomaly Detection

Anomaly detection lets you define the boundaries of what you consider to be good performance in your application and notifies you when and where metrics go over their limits.

Create custom triggers to alert you when performance dips, background job count increases by 20%, or memory suddenly increases.

AppSignal gives you the data-driven confidence to act proactively instead of urgently reacting to issues in your application.

Host Monitoring

AppSignal also monitors your Python application's hosts, giving you at-a-glance access to CPU, disk, network, and memory metrics. This allows you to understand how your Python application is performing beyond just looking at its code. Effectively manage and monitor your hosts.

Python Dashboards

AppSignal's dashboards give you instant visual insights into your application's metrics, allowing you to track and trace performance metrics quickly.

If you see something you want to investigate further, click on that point on the chart, and you'll view the state of your application at that specific point in time. Add custom markers to help you and your team better understand how your application performs.

More Than Just Monitoring

AppSignal can ingest and store logs from popular platforms such as AWS Kinesis, Heroku, and Netlify or via our dedicated logging API endpoint. Gain deep insights into your code's performance by combining your APM and log management.

Our intuitive UI takes you from an error message to a log line in just a few clicks and lets you query, filter, and share logs effortlessly.

Effortless Setup, Immediate Impact

You can have your Python application push metrics to AppSignal in less time than it takes to drink a coffee.

Sign up for an AppSignal account and follow our installation wizard instructions. The installation wizard will walk you through all the steps needed to send metrics from your Python application to AppSignal!

Our Python documentation will also take you through all the steps required to get the metrics you need, including how you can install AppSignal manually.

AppSignal: Your Python APM Tool

AppSignal is built by developers for developers to give you the visibility and insights needed to manage and scale your Python application effectively. AppSignal offers:

A 360-degree view of your application data.
An intuitive interface that's easy to navigate.
A clean approach to connecting errors, logs, and performance incidents.
The ability to monitor your Node.js, front-end JavaScript, Ruby, Elixir, and Python applications in one place.

If you're a new trial user, reach out to us once you've got your Python application set up with AppSignal, and we'll send a package of stroopwafels to you 🍪!

Python Wizardry by AppSignal

Monitoring Django Query Performance with AppSignal

Prerequisites

Setting Up AppSignal in a Django Project

How AppSignal Tracks Database Queries

Identifying Slow Queries in the AppSignal Dashboard

Common Django Query Problems and How to Spot Them

The Good Old N+1 Queries

Missing Database Indexes

Unbounded QuerySets

Repeated Identical Queries

Next Steps and Resources

Tracing a Slow Request Through Your Django App

Setting up the Demo Application

Installing AppSignal

Creating the Data Models

Populating the Database

Creating a Slow Django Endpoint

Triggering the Slow Request

Step 1: Finding the Slow Endpoint

Step 2: Analyzing Request Performance

Understanding the Root Cause

Fixing the Slow Query

Comparing the Results

A Repeatable Debugging Workflow

Why Observability Matters

Conclusion

Tracking Celery Task Failures in Python

Celery as a Default for Background Jobs in Python

The Problem with Celery’s Defaults

Celery Runs, Fails, and Retries

Exceptions Go to Logs Nobody Watches

Retries Hide Real Failures

You Find Something Is Broken From Users

What AppSignal Captures

Automatic Task Instrumentation

Exceptions With Full Tracebacks

Retry Visibility

Task Context and Arguments

Installing AppSignal

Seeing Failures as They Happen

Celery Task Failures Appear in the Errors Dashboard

Inspecting the Error Details

Why This Matters in Production Systems

Patterns to Watch For

Retry Storms

One Task Dragging Down the Queue

External Dependency Failures

Alerting on What Matters

Error Rate Spikes

New Exception Types

Task Duration Anomalies

Example: Debugging a Silent Failure

Wrap-Up

Improve Query Performance Using Python Django QuerySets

The Importance of Database Performance in Web Applications

Impact on User Experience

Impact on Server Resources

Understanding Django QuerySets

What is a QuerySet?

Key Characteristic of QuerySets: Laziness

Why Use Django QuerySets?

Implementing Basic QuerySet Performance Techniques

Requirements, Dependencies, and Settings

Setting up a Sample Model

Fetching Specific Fields with values() and values_list()

Efficient Counting with count()

Efficient Existence Checks with exists()

Wrapping Up

Switching from Pip to uv in Python: A Comprehensive Guide

What is uv?

Traditional Python Package Approach: pip and virtualenv

Installing uv

Managing Virtual Environments with uv for Python

Replacing Pip with uv for Dependency Management

Managing Python Versions with uv

Simplifying Tasks with uv

Creating and Managing Python Projects with uv

Managing Dependencies to a uv Project

Upgrading and Removing Dependencies

Fetching Specific Fields with `values()` and `values_list()`

Efficient Counting with `count()`

Efficient Existence Checks with `exists()`

Traditional Python Package Approach: `pip` and `virtualenv`

Step 9: Fix the N+1 Query Problem With `select_related()`

Step 10: Fix the N+1 Query Problem With `prefetch_related()`

When to Use `prefetch_related()` or `select_related()`