Add a Processing Progress Bar to Your Site

If you find the need to use progress bars on your site, I highly recommend using celery to queue your processes (so your system isn't overloaded). Building Progress Bars for the Web with Django and Celery is a great tutorial. Unfortunately for me, celery does not support Windows, so I had to come up with another solution.

First, make the HTML elements:

<div style="height: 500px" class="w3-container w3-padding-64 w3-theme-l5" id="contact">
    <div style="height: 50px"></div>
    <form method="post" class="w3-container w3-card-4 w3-display-top w3-padding-16 w3-white" id="form" enctype="multipart/form-data">
        {% csrf_token %}
        <div id="inputs">
            <div class="w3-row">
                <div class="w3-quarter">
                    <p class="w3-padding-16">File: <input type="file" name="myfile" id="myfile"> </p>
                </div>
            </div>
        </div>
        <div id="myProgress">
          <div id="myBar">0%</div>
        </div>
        <div class="w3-quarter">
            <input type="button" name="upload" value="upload" onclick="upload_file()">
        </div>
    </form>
</div>

Add some styling:

#myProgress {
  width: 100%;
  background-color: grey;
}

#myBar {
  width: 0%;
  height: 30px;
  background-color: #4CAF50;
  text-align: center; /* To center it horizontally (if you want) */
  line-height: 30px; /* To center it vertically */
  color: white;
}

Note that the upload button is of type button not submit so the submission can be handled in the javascript.

That was the easy part. For me, I can approximate the final file size and check the progress based on that, so I need to keep the path to the file hanging around. To do that, that means I need to submit the file(s) with AJAX so I can get the file path returned.

The AJAX upload looks like this:

    upload_file = function () {
        var formData = new FormData();
        formData.append('myfile', $('#myfile')[0].files[0]);
        formData.append('csrfmiddlewaretoken', '{{csrf_token}}');

        $.ajax({
           url : '',
           type : 'POST',
           data : formData,
           processData: false,
           contentType: false,
           success : function(data) {
              check_progress(data['rel_path']);
           }
        });
    };

You can see above that we are building a form and appending the file(s) and csrf_token to it when the upload button is pressed. The data is submitted to the same view as out GET request for the page:

def upload(request):
    if request.method == 'POST':
        file = request.FILES.get('myfile')
        upload_id = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(20))
        rel_path = os.path.join(datetime.now().strftime('%Y%m%d'), upload_id)

        fs = FileSystemStorage(location=os.path.join(MEDIA_ROOT, rel_path))
        fs.save(file.name, file)
        file_rel_path = os.path.join(rel_path, file.name)
        threading.Thread(target=process_file, args=(file_rel_path, )).start()
        return JsonResponse({'status': 'Processing...', 'rel_path': file_rel_path})

    else:
        return render(request, "contributor/upload.html")

To keep this non-blocking without celery, a new thread was created to do the processing:

threading.Thread(target=process_file, args=(file_rel_path, )).start()

The function process_file does the processing (not this is a simple dummy example):

def process_file(file_rel_path):
    file_path = os.path.join(MEDIA_ROOT, file_rel_path)
    print('async start')

    for i in range(100):
        file = open(file_path, 'a')
        file.write('line\n')
        file.flush()
        os.fsync(file.fileno())
        print('wroteline')
        time.sleep(1)
        file.close()

Note the arguments are passed with args=(file_rel_path, ) in the thread creation instead of passing it in with the function.

Finally, we are returning the path to the file. You can return anything you want to check the status. Another option is to update the database or something like redis on the progress and pass back the pk:

 return JsonResponse({'status': 'Processing...', 'rel_path': file_rel_path})

After a successful return in the upload_file function, we now that we have the path in the front end so we can start checking the progress by calling the function and passing the file path. Here it is again:

    upload_file = function () {
        var formData = new FormData();
        formData.append('myfile', $('#myfile')[0].files[0]);
        formData.append('csrfmiddlewaretoken', '{{csrf_token}}');

        $.ajax({
           url : '',
           type : 'POST',
           data : formData,
           processData: false,
           contentType: false,
           success : function(data) {
              check_progress(data['rel_path']);
           }
        });
    };

The check_progress function does a GET request to the "progress" URL and passes the data along:

    check_progress = function (path) {
        $.ajax({
           url : 'progress',
           type : 'GET',
           data : {file_path: path},
           success : function(data) {
               setTimeout(function () {
                   check_progress(path);
                    console.log(data['progress']);
                    move((data['progress']));
               }, 5000);
           }
        });
    };

If you look at the url, you will see GET '/upload/progress?file_path=20190412%2FH1Z61RQ4VIJD6BALX2N8%2Fresult.txt'

Now, create the view:

def check_progress(request):
    file = request.GET.get('file_path')
    parts = re.split('/', file)
    path = ''
    for part in parts:
        path = os.path.join(path, part)
    # calculate estimated complete file size
    c_size = 500    # bytes
    # check size of file
    file = os.path.join(MEDIA_ROOT, path)
    size = os.path.getsize(file)
    progress = (size/c_size * 100)//1
    print(size)
    print(c_size)
    print(progress)
    return JsonResponse({'status': 'Processing...', 'progress': progress})

Update urls.py:

path('upload/', views_contrib.upload, name='upload'),
path('upload/progress', views_contrib.check_progress, name='progress'),

To check the progress every 5 seconds, we can hit that url requesting the progress of that file:

check_progress = function (path) {
    $.ajax({
       url : 'progress',
       type : 'GET',
       data : {file_path: path},
       success : function(data) {
           setTimeout(function () {
                check_progress(path);
                console.log(data['progress']);
                move((data['progress']));
           }, 5000);
       }
    });
};

Here we are passing the file path back to the view, getting a response with the % complete, waiting 5 seconds, calling the move() function to set the progress bar, and then repeating (note, code needs to be added to stop requesting the progress when it's complete).

Finally, to set the progress bar, we pass the percentage complete to the move() function:

function move(width) {
    console.log('moce');
    console.log('progress ' + width);
    var elem = document.getElementById("myBar");
    elem.style.width = width + '%';
    elem.innerHTML = width * 1 + '%';
}

I hope that was easier to follow than it was to write.

April 12, 2019, 9:34 a.m.
Load Comments