Most of us might have faced the problem of our Django views running too slow with response times taking longer than 10 sec. Sometimes we might also get a HTTP 408 error meaning "The request was timed out" which makes the user experience worse.

In some cases it is possible to optimize your code so that it becomes more efficient ultimately reducing the response time but what to do in cases where that is not possible.

Let's go through an example:

def my_sample_django_view(request):

    makethirdpartycalls() # Takes >10s
    some_extra_processing() # Takes >10s
    return JsonResponse({"success":True}, status=HTTP_200_OK)

You can run into such kind of situations when

• There are multiple calls to third party APIs required (ex: stripe or sendgrid)

• You have to run some complex database queries

• Perform some other computationally extensive task

In any of the above sceenarios, you cannot reduce the time beyond a certain point. So, what is the solution to this problem?

The solution is to

1. Identify the tasks which are taking too much time

2. Push the execution of those tasks to the background.

By default, throughout the request lifecycle in Django everything runs on a single thread. This means a line of code cannot execute until the lines before have been executed. To make our views faster, we can shift our time taking tasks to a different thread, so our Django main thread doesn't have to wait for the time taking task to complete before it can return a response. This avoids the risk of the user getting a 408 response.

Here is an example:

def my_sample_django_view_updated(request):

    shift_task_elsewhere(makethirdpartycalls, some_extra_processing) # Takes 0.01s

    return JsonResponse({"success":True, "message": "your task will be completed soon"}, status=HTTP_200_OK)

This can be achieved with a lot of tools/packages available today with Django. For example:

1. Django Celery

2. huey

3. Django Background Tasks

Note: You can also use Python's threading module to perform some tasks asynchronously. It may or may not improve performance due to the GIL limitation in Python, but in some cases it might be easier and more transparent to use threading.

The tools / packages mentioned above works by either storing your long running tasks in a database (Django-background-tasks, huey) or a dedicated messaging queue like Redis or RabbitMQ (Django Celery) and then they spawn a worker thread (separate from the Django main thread) to works on the tasks in the background.

Django Celery is the most popular of them all and is very extensive and reliable but in most cases it might be overkill because it requires a lot of setup and configuration. Below, I will be discussing setting up and using django-background-tasks which is a breeze to setup and use and works well for most of the cases.

Setting up Django Background tasks

Install Django-background-tasks to your Django project folder using pipenv as the virtual environment.

cd [my-django-proect]
pipenv install django-background-tasks

Now add 'background_task' to INSTALLED_APPS in settings.py:

#settings.py file
INSTALLED_APPS = (
    'background_task',
    # ...
)

then perform database migrations to ensure the django-background-tasks schema is in place:

pipenv shell
(my-django-project)$  python manage.py migrate

Finally, we need to run the django-background-tasks process. This spawns the worker process which will execute our long running tasks. To run this process, open a new terminal window and start the process via the Django management script:

python manage.py process_tasks

Real world example

Let's say you are working on a project which involves customers making payments on your website and after the payment is successful you need to call some 3rd party APIs to register the user to their site and then save the registration id and then finally create the order in your database.

Your existing django code would look something like this:

#models file

class Order(models.Model):
    third_party_order_id = models.CharField(max_length=16)
    price = models.DecimalField(decimal_places=2, max_digits=7)
    user = models.ForeignKey(User, on_delete=models.CASCADE)

# views.py

def create_order(request):
    """
    creates a new order
    """
    user = request.user
    price = fetch_payment_amount_from_stripe(user) #Takes 5-6s
    third_party_order_id = create_order_on_some_3rd_party_sites(user) #Takes 10-20s
    order = Order.objects.create( price=price, user=user, third_party_order_id=third_party_order_id)


    return render(request, 'orders.html', {'order': order})

Now let's try to improve the above code by using django-background-tasks:

We can start by moving the slow code into a separate task function, which will be run by the background task worker. We can also put this in a separate tasks.py file for better structure.

# task.py
from background_task import background

@background(schedule=1)
def create_order_task(user):

   price = fetch_payment_amount_from_stripe(user) #Takes 5-6s
   third_party_order_id = create_order_on_some_3rd_party_sites(user) #Takes 10-20s
   Order.objects.create( price=price, user=user, third_party_order_id=third_party_order_id)

Note: Please note that the user (argument to the create_order_task function) should be JSON serializable because django-background-tasks stores this function in a database as a string.

The background decorator at the top defines after how much time of the function getting called upon is the actual task going to run. It is important to note that calling the function does not actually execute its code; rather a Task record is stored into the database by the "django-background-tasks" module, more precisely into the "background_task" table.

Now we've separated the long running task, we will just call it from our view:

# views.py
from .tasks import create_order_task

def create_order(request):
    """
    creates a new order
    """
    user = request.user
    create_order_task(user) #Task is scheduled instantly by dispatching to background worker


    return render(request, 'orders.html', {'message': "order creation is scheduled"})

Previously the user clicks the "Place Order" button, waits for 20 seconds and then finally gets their newly created order id but now although the user gets a response much faster their order isn't created yet. To improve this situation further we can add a loading state to our order model and create the order without the fields that are dependent on the background task. Let's try this:

#models.py updated
class Order(models.Model):

    third_party_order_id = models.CharField(max_length=16, blank=True, null=True)
    price = models.DecimalField(decimal_places=2, max_digits=7, null=True)
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    is_loading = models.BooleanField(default=True)

Then in the view we create the Order object without the background dependent fields and then return the pending order with is_loading set to True.

# views.py updated
from .tasks import create_order_task

def create_order(request):
    """
    creates a new order
    """
    user = request.user
    create_order_task(user) #Task is scheduled instantly by dispatching to background worker
    order = Order.objects.create( user=user)

    return render(request, 'orders.html', {'order': order})

Finally, we can set the Order state back to "not loading" when the order has all the required fields:

# task.py updated
from background_task import background

@background(schedule=1)
def create_order_task(user):

   price = fetch_payment_amount_from_stripe(user) #Takes 5-6s
   third_party_order_id = create_order_on_some_3rd_party_sites(user) #Takes 10-20s
   order = Order.objects.filter(user=user)
   order.third_party_order_id = third_party_order_id
   order.is_loading = False
   order.save()

Now the user can see the order immediately created but in a pending state and when they refresh the page after a while they can see the order creation process complete.