Uploading Files to AWS S3 with Python and Django

Introduction

In the quest to build more interactive websites, we don't only relay information to users but also allow them to upload data of their own. This opens up more opportunities and more ways that our websites can serve the end-users.

By allowing users to upload files, we can allow them to share photographs, videos, or music with others or back them up for safekeeping. We can also provide the functionality to manage files and convert them into other formats through websites instead of installing native apps.

The rise of social media globally can be attributed to the ability of users to upload their files, mostly in the form of images and videos for other users to see and also as a means of communication. By enabling users to upload files to websites and platforms, means of communication have been enhanced and information can now be spread in many different formats.

In this post, we will explore how Django handles file uploading and how we can tap into and extend this functionality with cloud storage to suit our needs.

How Django Handles File Storage

Django not only allows us to turn concepts into web applications but also provides functionality for us to handle files and allow users to upload files to our web applications for further interaction. Through forms, users can attach files to their requests and have their files uploaded and stored in our backend servers.

Before a file is saved, it is temporarily stored somewhere before being processed and stored in the intended final location. For instance, if the uploaded file is less than 2.5MB, the contents of that file will be stored in memory, then written to disk once all the operations are completed while processing it.

This makes the process fast for small files. For files larger than 2.5MB, they are first written to a temporary location as the data is being received, then once processing is complete, the file is moved to its final destination.

File behavior in Django can be customized through various settings, such as FILE_UPLOAD_MAX_MEMORY_SIZE, which allows us to modify the 2.5MB upload limit size for the files that are written to memory first and not to a temporary location. We can also configure the default permissions for the uploaded files through the FILE_UPLOAD_PERMISSIONS.

Other settings can be found in this section of the official Django documentation.

Where can we store our files?

In a Django-powered web application, we can store the uploaded files in various different locations. We can store them on our own servers where the Django code is deployed, or we can send them over to other servers which may have been set up elsewhere for storage purposes.

In a bid to cut server maintenance costs and enhance performance, we can also choose not to store the uploaded files on our own servers. In this case, we can hand them over to other hosted storage providers such as AWS, Azure, or OneDrive, among others.

There are several packages that allow us to interact with the APIs provided by the various service providers that we have mentioned. They include:

• Django-Storages enables us to store our uploaded files to AWS Simple Storage Service (S3), Dropbox, OneDrive, Azure and Digital Ocean.
• Django-S3-Storage through which we can upload files directly to Amazon S3
• Django-Cumulus, which allows us to interact with Rackspace for our storage need
• Others include Django-Dropbox, Django-Storage-Swift, and Django-Cloudinary-Storage. More can be found here

For this post, we will use the Django-s3direct package to store our files on AWS's S3.

Our Application - Django Drive

We will use Django to build a web application in which we will upload content for end-users to view. This will be achieved by using the Django administration interface, which comes with the framework.

Our site will be used to sell cars and on it, we will display details and add images or videos of the cars on sale.

The images or videos of the cars on sale will be stored on S3. We will not implement user registration or login at this time for brevity.

Setup

We will use Pipenv to set up and manage our isolated environment in which we will build our Django application by running the following command to set it up using Python3:

$ pipenv install --three

With the environment set up, we can now install Django and Django-s3direct to handle our file uploads to S3:

$ pipenv install django django-s3direct

Django provides a set of commands to bootstrap our project before we start implementing the core functionality of our application. Our Django drive project will have a single application that will be the focus of this post. To achieve this, we run the following commands:

$ django-admin startproject django_drive && cd django_drive
$ django-admin startapp django_drive_app

The django-admin startproject ... command creates the project, and the django-admin startapp ... command creates the application.

The last step of our setup is to create database tables by running the migrate command:

$ python manage.py migrate

When we start our project by running the command python manage.py runserver, we are welcomed by the following page, which confirms that our setup was successful:

Since we will be uploading our files to AWS S3, we will need to set up a free-tier AWS account for demo purposes. After setting up, we can navigate to the S3 dashboard and create a new bucket that will contain our uploads.

For Django-s3direct to interact with our AWS setup, we need to provide the following credentials AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and the AWS_STORAGE_BUCKET_NAME.

Next, we will add the following to our django_drive/settings.py file:

AWS_ACCESS_KEY_ID = 'aws-access-key-id'
AWS_SECRET_ACCESS_KEY = 'secret-access-key'
AWS_STORAGE_BUCKET_NAME = 'name-of-the-bucket'
AWS_S3_REGION_NAME = 'name-of-the-region'
AWS_S3_ENDPOINT_URL = 'https://s3.amazonaws.com'

S3DIRECT_DESTINATIONS = {
    'primary_destination': {
        'key': 'uploads/',
        'allowed': ['image/jpg', 'image/jpeg', 'image/png', 'video/mp4'],
    },
}

Django-s3direct allows us to specify more than one destination for our uploads, this way we can direct different files to separate S3 buckets. For this project, we will put all the uploads in one bucket. Another nifty feature is that we can also limit the file types that can be uploaded to our website. In our case, we have limited it to MP4 videos, JPEG, and PNG images only.

Note: More details about setting up Django-s3direct, such as CORS and Access Setup, can be found here.

We also need to add the following entries in the django_drive/urls.py file:

from django.urls import path, include

urlpatterns = [
    ...
    path('', include('django_drive_app.urls')),
    path('s3direct/', include('s3direct.urls')),
    ...
]

Implementation

We will start by creating the model for our car data, which will be displayed to the end-users. This model will also define the information we will input in our admin dashboard when adding cars to our platform. The car model will be as follows:

from django.db import models
from s3direct.fields import S3DirectField

class Car(models.Model):
    name = models.CharField(max_length=255, blank=False, null=False)
    year_of_manufacture = models.CharField(max_length=255, blank=False, null=False)
    price = models.CharField(max_length=255, blank=False, null=False)
    image = S3DirectField(dest='primary_destination', blank=True)
    video = S3DirectField(dest='primary_destination', blank=True)

    def __str__(self):
        return f"{self.name} ({self.year_of_manufacture}) - {self.price}"

For each car, we will store its name, year of manufacture, price, and an image or video. After creating the model, let us make migrations to create the table in the database that will hold our data by running:

$ python manage.py makemigrations
$ python manage.py migrate

Since we will be using the Django admin dashboard to manage the cars on our platform, we need to register our model in the django_drive_app/admin.py:

from django.contrib import admin
from.models import Car

admin.site.register(Car)

Then we need to create the superuser who will be in charge of adding the cars by running the following command and following the prompts:

$ python manage.py createsuperuser
$ python manage.py runserver

The python manage.py runserver command simply restarts our application.

After restarting our server, we can now navigate to the administration dashboard at http://127.0.0.1:8000/admin and log in with the credentials we specified earlier. Under site administration, we can see our DJANGO_DRIVE_APP with the option to add or change existing cars.

This is the form we use to add a car and its details:

Once we save our car, we can find the image we have uploaded in our S3 bucket on the AWS console. This means that our file has been uploaded to AWS.

Now we will create a view to display the cars and their data to the end-users of our website and also display the images or videos associated with each car. We will start by creating a view in the django_drive_app/views.py:

from django.shortcuts import render
from django.views.generic import TemplateView
from .models import Car

class CarView(TemplateView):
    template_name = 'django_drive_app/cars.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['cars'] = Car.objects.all()
        return context

In this view, we use a class-based Django view to render the HTML file to display our cars. In our view, we run a query to fetch all the cars as stored in our database.

Next, let us create django_drive_app/templates/django_drive_app/cars.html to render our cars:

<!DOCTYPE html>
<html>
  <head>
    <title>Django Drive</title>
  </head>
  <body>
    <h3>Welcome to Django Drive. </h3>
    <p>Here are the current cars available for sale: </p>
    <div class="cars-container">
      {% for car in cars %}
        <div class="car">
          <p>
            <b> {{ car.name }} ({{ car.year_of_manufacture }}) </b> <br>
            Price: {{ car.price }}
          </p>
          <!-- if the car has an image attached -->
          {% if car.image %}
          <img src="{{ car.image }}" height="200" width="400"/>
          {% endif %}
          <!-- If the car has a video -->
          {% if car.video %}
            <video width="320" height="240" controls>
                <source src="{{ car.video }}" type="video/mp4">
              Your browser does not support the video tag.
            </video>
          {% endif %}
        </div>
        <hr>
      {% endfor %}
    </div>
  </body>
</html>

With the view and the template in place, let us add the endpoint that will be used to display the list of cars to the end-users by creating the django_drive_app/urls.py:

from django.conf.urls import url
from .views import CarView

urlpatterns = [
  url(r'^cars/$', CarView.as_view(), name="cars"),
]

We import our view and add a URL entry to map the endpoint to the view that will render the cars. When we restart our server and navigate to 127.0.0.1:8000/cars/, we encounter the following:

As we can see, we created cars with attached images and videos and had them uploaded to AWS's S3 service. The same images and videos have been rendered in our web application after being fetched from AWS.

Conclusion

In this article, we have created a simple Django application that allows administrators to upload files to AWS S3 through the Django administration dashboard. We rendered the uploaded files as hosted on S3 on our landing page, including videos and images of the cars that users would wish to purchase or view.

We used the Django-s3direct library to handle the interaction between our Django application and AWS S3 where our files are stored. Through the Django administration application, we were able to upload files that were eventually rendered to the end users on our landing page. We were able to upload and render both images and videos.

The source code for this project is available here on GitHub.