Categories
Uncategorized

How to search CloudWatch logs for Metrics (or errors) on lambda

This post is a simple step by step guide on how to search Cloudwatch for errors on a lambda function. I find that this simple task can be confusing sometimes, at least for me, so I’m sharing what I do about this that helps me, enjoy 🙂


Steps to search:
– Go to AWS console
– Under services, select ‘Lambda’
– Search for and select the desired function name on the list of available functions
AWS Lambda function search page

– on the function page, select the ‘Monitoring’ tab


– on the ‘Monitoring’ tab, click on ‘view logs in Cloudwatch’
View Lambda cloudwatch logs

– on the CloudWatch page that you will be redirected to from the lambda page, select a log group, targeting the time when the error was encountered

NB: take note of the time zones, between when you received the error and the time of the location you are accessing/running AWS from and convert accordingly so you know what time to target for the search.

AWS lambda CloudWatch log group

– After selecting the time frame from the previous step, you land on the log group page.
Here you can search for the desired metric. This can be an AWS lambda metric or custom metric logged by your code.
In this post, we will search for a custom metric ‘filtered’.

This string can also be “error” or any string of interest.

A search for ‘filtered’ will show you the errors that happened during the time of interest selected.

– Click on any of the results to view the details of the error or the log.

AWS ClouWatch log group

AWS ClouWatch log group


Any of these metrics can be searched for on AWS lambda
https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html



Categories
code Django git python

Django management commands from admin interface

This need came up when during app deployment, needed to run some management commands without ssh-ing into the server or using the terminal.
Thankfully Django management command can be called anywhere using management.call_command()

This is a brief one, view code on GitHub here:
https://github.com/NEbere/data-import/tree/master

You need to install the admin_plus package with pip, ensure your virtual environment is setup and activated if that’s what you’re using, in any case, ensure you have the right environment setup and running for your app before installing to avoid any issues 😉

Install admin_plus:

pip install django-adminplus

Add adminplus to INSTALLED_APPS in your settings file

If you’re using Django 1.7, you should also replace django.contrib.admin with django.contrib.admin.apps.SimpleAdminConfig in your installed apps, in order to disable the automatic auto-discovery:

so update ‘django.contrib.admin’ to ‘django.contrib.admin.apps.SimpleAdminConfig’ in INSTALLED_APPS in your settings file

import AdminSitePlus in urls.py of your main package, the file where URL’s from installed apps are registered
hint: this is where you have the admin URL defined

In that same file, add this for the admin_plus

admin.site = AdminSitePlus()
admin.autodiscover()

Now, head to the admin.py file in your app and call the command you want to execute.

from django.contrib import admin
from .models import Movie
from django.core import management
from django.shortcuts import redirect

class MovieAdmin(admin.ModelAdmin):
    @admin.site.register_view('import_movies_from_url', 'Import Movies from URL')
    def import_movies_from_url(request):
        print('import movies here')
        try:
            management.call_command('import_from_url')
            message = 'successfully imported data from URL'

        except Exception as ex:
            message = 'Error importing from data from URL {}'.format(str(ex))

        admin.ModelAdmin.message_user(Movie, request, message)
        return redirect('admin:index')

admin.site.register(Movie, MovieAdmin)


the command I am calling here is import_from_url and that name corresponds to the file where I defined the command (minus the file .py file extension)

This does two things:
1. Makes your Model available on the admin interface, you can create, update, delete models objects from the admin interface once your models are registered.
2. Provides a link that when clicked runs the management command defined to be run.

Screen Shot 2017-10-26 at 1.36.02 PM

More details on admin_plus

Again, here is the github repo with all the import scripts and sample files:
https://github.com/NEbere/data-import/blob/call-management-command-from-admin/
Happy coding 🙂

Categories
code git

Import data into database – django app

Steps to import data into your Django app

So you have a Django app and want to load data from a URL, a CSV file or a JSON file, I got you 🙂
First, create your models, make migrations, migrate and all that setup you have to do, but I’m sure you got all that covered 😉

See it on Github here: https://github.com/NEbere/data-import/tree/master

Import from URL:

This code gets data from a URL, be sure to have requests installed or any other HTTP library of choice.

"""
Import json data from URL to Datababse
"""
import requests
import json
from import_data.models import Movie #Import your model here
from django.core.management.base import BaseCommand
from datetime import datetime

IMPORT_URL = 'https://jsonplaceholder.typicode.com/photos' # URL to import from

class Command(BaseCommand):
  def import_movie(self, data):
  title = data.get('title', None)
  url = data.get('url', None);
  release_year = datetime.now()
  try: #try and catch for saving the objects
    movie, created = Movie.objects.get_or_create(
    title=title,
    url=url,
    release_year=release_year
    )
    if created:
      movie.save()
     display_format = "\nMovie, {}, has been saved.
     print(display_format.format(movie))
  exceptExceptionas ex:
    print(str(ex))
    msg = "\n\nSomething went wrong saving this movie: {}\n{}".format(title, str(ex))
    print(msg)
  def handle(self, *args, **options):
  """
  Makes a GET request to the API.
  """
  headers = {'Content-Type': 'application/json'}
  response = requests.get(
  url=IMPORT_URL,
  headers=headers,
  )
  response.raise_for_status()
  data = response.json()
  for data_object in data:
  self.import_movie(data_object)

Import from files (JSON):

To import from a file, read the file location

import os # Add os import to the imports used above for URL
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) #Define BASE_DIR or import of it is previously defined
class Command(BaseCommand):
  def import_movie_from_file(self): 
   data_folder = os.path.join(BASE_DIR, 'import_data', 'resources/json_file')
   for data_file in os.listdir(data_folder):
     withopen(os.path.join(data_folder, data_file), encoding='utf-8') as data_file:
       data = json.loads(data_file.read())
       for data_object in data:
         title = data_object.get('title', None)
         url = data_object.get('url', None)
         release_year = datetime.now()
         # Use the try and catch used above here as it's the same model
 def handle(self, *args, **options):
 """
 Call the function to import data
 """
 self.import_movie_from_file() # call the import function in handle

Import from files (CSV):

To import from a csv uses the same steps defined for JSON file import above with the difference been importing csv, and how the data is parsed.

import csv # Add csv import to the imports used above for JSON import
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) #Define BASE_DIR or import of it is previously defined
class Command(BaseCommand):
  def import_movie_from_csv_file(self): 
   data_folder = os.path.join(BASE_DIR, 'import_data', 'resources/csv_file') # folder where csv file is stored
   for data_file in os.listdir(data_folder):
     withopen(os.path.join(data_folder, data_file), encoding='utf-8') as data_file:
       data = json.loads(data_file.read())
       for data_object in data:
         title = data_object[1] # the value at index 0 is the ID and we dont need that here
         url = data_object[2]
         release_year = datetime.now()
         # Use the try and catch used above here as it's the same model
 def handle(self, *args, **options):
 """
 Call the function to import data
 """
 self.import_movie_from_csv_file() # call the import function in handle

Run the import commands

 python manage.py import_from_url # import_from_url is the name of the file where the command is defined without the extension, that is, .py

for more about requests http://docs.python-requests.org/en/master/

Again, here is the github repo with all the import scripts and sample files:
https://github.com/NEbere/data-import/tree/master
Happy coding 🙂

Categories
Uncategorized

Effective collaboration and communication

So I did this small talk on effective collaboration and communication where I work so I thought I’d share.

[pdf-embedder url=”http://gelcnigeria.org/devChannel/wp-content/uploads/2017/05/05-05-2017-Effective-Collaboration-and-communication-etiquettes-in-Software-development-By-Happiness-Nwosu-.pdf”]

Download here

Categories
bower install git Hacks

Bower install from a forked repository

If for any reason you need to fork a repo and want to install your forked version using Bower, follow these steps:

  1.  Fork the repository
  2. Clone the forked repo on your computer
  3. Make desired changes
  4. Increment version number in bower.json (create one if none exists e.g. 4.0.1)
  5. Commit and push
  6. Create a new version tag higher than that of the forked repository. If version tag from respository is 4.0.1, increase it. e.g. 4.0.2 :
     $ git tag "4.0.2" 
  7.  push tag :
    $ git push --tag 
  8. Install in your app using bower:
     $ bower install https://github.com/yourName/forkedRepository.git#4.0.2 
Categories
Fixes

Update a model field – Django

I noticed a typo in a field name after running migrations, I followed these steps to update the field without loosing data.

  1. Update the filed name in the model with the desired name (taking note of the current name)
  2. Create an empty migration. If you have previous migrations, this one must be named in such a way that it comes last in the migrations list. eg if you already have 001_users and 002_emails then this new migration needs to be named 003_new_name

The new migration should look like this:


from __future__ import unicode_literals

from django.db import migrations
class Migration(migrations.Migration):

dependencies = [
('django_app_name', 'last_previous_migration'), # last_previous_migration is the migration immediately before this new empty migration without the .py extension
]

operations = [
migrations.RenameField(
model_name='model_name',
old_name='old_filed_name', #that was taken note of from step one above
new_name='description', #the new/desired field name
),
]

3. run migrations

 $ python manage.py migrate 

This will update the field.

Hope this saves someone some googling time



		
Categories
Fixes Hacks

virtualenv for python3

Simple steps to setup virtualenv for Python3

To setup virtualenv for python3 for Django projects, create and cd into the desired folder for the Django project.

Create a folder for virtualenv, eg mkdir venv

$ cd venv // The created folder for virtualenv 

Then run these commands worked for me in this order:

$ virtualenv -p python3 . // setup virtualenv with python3 fin current directory 
$ source bin/activate // ativate virtualenv 

At this point, checking python version will give python3

Install Django:

$ pip install django 

To see all installations run :

$ pip freeze //Django should be included in the list of installations 

Start Django project

$ django-admin startproject --projectname 

cd into created project and run:

$ python manage.py runserver 

More on virtualenv

 

 

Categories
Fixes Hacks

Fix IOError: [Errno 2] No such file or directory error for appdirs

Appdirs is a small Python module for determining appropriate platform-specific dirs, e.g. a “user data dir”.

Github repo – https://github.com/ActiveState/appdirs

This error from my experience occurred when I updated my local branch from origin and ran pip install to get the latest installed packages.

pip tried to upgrade appdir but it failed for some unknown reasons, then the error came up:

IOError: [Errno 2] No such file or directory: ‘/.local/lib/python2.7/site-packages/appdirs-1.4.0.dist-info/METADATA’

What worked for me was running

$ pip install appdirs --upgrade 
then:
$ pip install -r requirements.txt --upgrade 

The issue was addressed here:

https://github.com/ActiveState/appdirs/issues/89