Leave no trace behind…

I am not suggesting you do this with any bad intention to “cover your tracks” and so no one is able to catch you. I recommend this for your learning/researching and most importantly, to think how to protect your environment from such activities.

To hide your activities in a Linux environment, you execute the following in the exact sequence:

No trace… pure evil… my question to you now is how do you stop someone from doing this to you, if you are the owner of your systems?

Leave no trace behind…

Create a REST API With Django

I am going to use Python and the Django REST API framework to build a simple web service that will be able to provide data and functionality to many different platforms and applications. I will be building an invoicing API, with endpoints for viewing, creating and deleting invoices. All within 20 steps or less. I am assuming you already have Python installed on your PC/Mac. I am using Mac but the steps shouldn’t be any different for PCs.

STEP 1
Create a folder somewhere in your local machine
Example:
$mkdir LearningREST

STEP 2
Create a virtual environment
Example:
$cd LearningREST
$virtualenv env

STEP 3
Activate the virtual environment
Example:
$source env/bin/activate

STEP 4
Install django in your virtual environment
Example:
$pip install django

STEP 5
Install django restframework
Example:
$pip install djangorestframework

STEP 6
Create a django project
Example:
$django-admin.py startproject invoicesproject

STEP 7
Create an app within the project
Example:
$cd invoicesproject
$python manage.py startapp invoices

Open the settings.py under invoicesproject.
Go to “INSTALLED_APPS” section,
add ‘rest_framework’
add ‘invoices.apps.InvoicesConfig’

STEP 8
Run the web server now
Example:
$python manage.py runserver

Check the line that says “Starting development server at”
Assuming the http address says http://127.0.0.1:8000/
Open a browser and browse to http://127.0.0.1:8000/

The message in the browser should say “It worked!”

STEP 9
Create a model class.
To do so, open the models.py under invoices
Enter the following information

class Invoice(models.Model):
created = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=100)
description = models.TextField(blank=True, default=”)
total = models.DecimalField(max_digits=7, decimal_places=2)
paid = models.DecimalField(max_digits=7, decimal_places=2)

Save the file. Open your terminal or the command prompt.
Execute the following
$python manage.py makemigrations invoices
$python manage.py migrate

You can test the model now from a shell.
In your terminal or your command prompt, execute the following

$python manage.py shell
>>> from invoices.models import Invoice
>>> Invoice.objects.all()
>>> invoice = Invoice(name = ‘My First Invoice’, description = ‘A very detailed description’, total = 100.00, paid = 0.00)
>>> invoice.name
>>> invoice.total
>>> invoice.paid
>>> Invoice.objects.all()
>>> invoice.save()
>>> Invoice.objects.all()

STEP 10

We will create a serializer class.
To do so, go to your invoices folder and create a file called serializers.py

Type the following content in the serializers.py

from rest_framework import serializers
from invoices.models import Invoice

class InvoiceSerializer(serializers.ModelSerializer):
class Meta:
model = Invoice
fields = (‘name’, ‘description’, ‘total’, ‘paid’)

Now we’ll try this serializer class from the shell.
Open your terminal or your command prompt and type the following:
$python manage.py shell
>>> from invoices.models import Invoice
>>> from invoices.serializers import InvoiceSerializer
>>> Invoice.objects.all()
>>> invoice = Invoice.objects.get()
>>> invoice.name
>>> serializer = InvoiceSerializer(invoice)
>>> serializer.data

STEP 11
We will create a View of our application.
To do so, open the views.py file under invoices folder and remove all the contents.

Type the follow in the views.py

from rest_framework.decorators import api_view
from rest_framework.response import Response
from invoices.models import Invoice
from invoices.serializers import InvoiceSerializer

@api_view([‘GET’])
def invoice_list(request):
if request.method == ‘GET’:
invoices = Invoice.objects.all()
serializer = InvoiceSerializer(invoices, many=True)
return Response(serializer.data)

STEP 12
Create a urls.py file under the invoices folder.
Type the following content

from django.conf.urls import url
from invoices import views

urlpatterns = [
url(r’^invoices/$’, views.invoice_list),
]

Open urls.py under invoicesproject folder and update with the following content

from django.conf.urls import url, include
from django.contrib import admin

urlpatterns = [
url(r’^’, include(‘invoices.urls’)),
]

STEP 13
Go to your terminal or your command prompt and execute the following
$python manage.py runserver

Go to the url http://127.0.0.1:8000/invoices/

STEP 14
Let's add the capability to retrieve a single invoice.
To do so, open the serializers.py under the invoice folder and update with the following

from rest_framework import serializers
from invoices.models import Invoice

class InvoiceSerializer(serializers.ModelSerializer):
class Meta:
model = Invoice
fields = (‘id’, ‘name’, ‘description’, ‘total’, ‘paid’)

Open the views.py file under the invoice folder and update with the following.
Add the following import

from rest_framework import status

Add the following method

@api_view([‘GET’])
def invoice_detail(response, pk):
try:
invoice = Invoice.objects.get(pk=pk)
except Invoice.DoesNotExist:
return Response(status = status.HTTP_404_NOT_FOUND)

if request.method == ‘GET’:
serializer = InvoiceSerializer(invoice)
return Response(serializer.data)

Open the urls.py under invoices and add the following:

url(r’^invoices/(?P[0-9]+)/$’, views.invoice_detail),

Execute this url http://127.0.0.1:8000/invoices/ in your browser.

Take the id number and append it to the url like the following

http://127.0.0.1:8000/invoices/1

In my case, the primary key for that record was 1

STEP 15
Let's add the capability to create a new invoice.
First, download postman to call your API from outside and post data to it. Download from here https://www.getpostman.com/

Open the views.py under the invoices folder and update the invoice_list function with the following

@api_view([‘GET’, ‘POST’])
def invoice_list(request):
if request.method == ‘GET’:
invoices = Invoice.objects.all()
serializer = InvoiceSerializer(invoices, many=True)
return Response(serializer.data)
elif request.method == ‘POST’:
serializer = InvoiceSerializer(data = request.data)

if serializer.is_valid():
serializer.save()
return Response(serializer.data, status = status.HTTP_201_CREATED)
return Response(serializer.errors, status = status.HTTP_400_NOT_FOUND)

Let’s restart our server to make sure all the changes took effect. Execute the following:

$python manage.py runserver

You can quickly check our invoice_list function with GET method is still working by browsing to

http://127.0.0.1:8000/invoices/

Now, let’s use postman to check this updated function fully.
Start postman, in the Builder tab (look at the top of this application to be sure. You should be there by default).

From dropdown where you see enter request URL, select POST and type the following
http://127.0.0.1:8000/invoices/

select Body right below it and select raw right below that. Select JSON from the body type. Enter the following in the body

{
“name”: “My newest Invoice”,
“description”: “Another description”,
“total”: 25.00,
“paid”: 25.00
}

Click Send and you should get a response back similar to the following.
The only difference between my example and yours could be the id value.

{
“id”: 2,
“name”: “My newest Invoice”,
“description”: “Another description”,
“total”: “25.00”,
“paid”: “25.00”
}

STEP 16

Let's add the update function in our API. To do so, open views.py.
Update the invoice_detail function with the following:

@api_view([‘GET’, ‘PUT’])
def invoice_detail(request, pk):
try:
invoice = Invoice.objects.get(pk=pk)
except Invoice.DoesNotExist:
return Response(status = status.HTTP_404_NOT_FOUND)

if request.method == ‘GET’:
serializer = InvoiceSerializer(invoice)
return Response(serializer.data)

elif request.method == ‘PUT’:
serializer = InvoiceSerializer(invoice, data = request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data)
return Response(serializer.errors, status = status.HTTP_400_BAD_REQUEST)

To test the new feature, open postman. Select PUT from GET or POST dropdown menu.
Type in the url http://127.0.0.1:8000/invoices/2/
We want to update the 2nd invoice
Select the body tab and check the raw body type. Make sure to select JSON as the message type.
Enter following content in the form body and click Send

{
“id”: 2,
“name”: “My awesome Invoice”,
“description”: “Another description”,
“total”: “2500.00”,
“paid”: “2500.00”
}

You should get a response back with your updated data.

STEP 17
Delete function is easier. Open views.py and update the invoice_detail function with the following content

@api_view([‘GET’, ‘PUT’, ‘DELETE’])
def invoice_detail(request, pk):
try:
invoice = Invoice.objects.get(pk=pk)
except Invoice.DoesNotExist:
return Response(status = status.HTTP_404_NOT_FOUND)

if request.method == ‘GET’:
serializer = InvoiceSerializer(invoice)
return Response(serializer.data)

elif request.method == ‘PUT’:
serializer = InvoiceSerializer(invoice, data = request.data)
if serializer.is_valid():
serializer.save()
return Response(serializer.data)
return Response(serializer.errors, status = status.HTTP_400_BAD_REQUEST)

elif request.method == ‘DELETE’:
invoice.delete()
return Response(status = status.HTTP_204_NO_CONTENT)

To test our delete function, let’s open postman. Select DELETE from the dropdown of GET, POST or PUT.
Provide the URL http://127.0.0.1:8000/invoices/2/
You shouldn’t see anything in the response body. The status should say, 204 No Content.
You have just deleted the record id 2.

To validate the record got deleted, you select the GET http method dropdown and type in the URL http://127.0.0.1:8000/invoices/

You should see only the remaining record.

Create a REST API With Django

Auto increment ID in Oracle

Unfortunately, there is no auto increment id in Oracle until 12c. So for the older versions, you can model it easily with a sequence and a trigger like the following example:

CREATE TABLE departments (
ID NUMBER(10) NOT NULL,
DESCRIPTION VARCHAR2(50) NOT NULL);

ALTER TABLE departments ADD (
CONSTRAINT dept_pk PRIMARY KEY (ID));

CREATE SEQUENCE dept_seq START WITH 1;

CREATE OR REPLACE TRIGGER dept_bir
BEFORE INSERT ON departments
FOR EACH ROW

BEGIN
SELECT dept_seq.NEXTVAL
INTO :new.id
FROM dual;
END;
/

For 12c, you can do the following:

CREATE TABLE t1 (c1 NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY,
c2 VARCHAR2(10));

CREATE TABLE t1 (
c1 NUMBER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),
c2 VARCHAR2(10)
);

Auto increment ID in Oracle

Installing Django from behind proxy

If you are sitting behind a proxy (like at your work place), you will get the following error trying to install Django.

Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by ‘ProxyError(‘Cannot connect to proxy.’, error(‘Tunnel connection failed: 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )’,))’: /simple/django/

To get around it, execute the following:

pip install django –proxy domain\username:password@proxyhost:port

Installing Django from behind proxy

Uninstall MySQL on a Mac

Steps to completely uninstall MySQL from a Mac is a lot more involved than I thought.

Here are the steps:

    Stop the database server
    sudo rm /usr/local/mysql
    sudo rm -rf /usr/local/mysql*
    sudo rm -rf /Library/StartupItems/MySQLCOM
    sudo rm -rf /Library/PreferencePanes/My*
    edit /etc/hostconfig and remove the line MYSQLCOM=-YES-
    rm -rf ~/Library/PreferencePanes/My*
    sudo rm -rf /Library/Receipts/mysql*
    sudo rm -rf /Library/Receipts/MySQL*
    sudo rm -rf /private/var/db/receipts/*mysql*
Uninstall MySQL on a Mac

Reset MySQL root password on Mac

Simple and clear way to update the root password MySQL on Mac:


1) sudo /usr/local/mysql/support-files/mysql.server stop
2) sudo /usr/local/mysql/support-files/mysql.server start --skip-grant-tables
3) cd to /usr/local/mysql/support-files/mysql
4) ./mysql> FLUSH PRIVILEGES;
5) mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';
6) Ctrl + z
7) sudo /usr/local/mysql/support-files/mysql.server start
8) /usr/local/mysql/support-files/mysql -u root -p
9) enter the new password i.e MyNewPass

Reset MySQL root password on Mac

Install Django on Mac

Execute the following command to install Django on Mac:

sudo pip install django

If the install is successful, you should see the following:



Collecting django
Downloading Django-1.10.4-py2.py3-none-any.whl (6.8MB)
100% |████████████████████████████████| 6.8MB 124kB/s
Installing collected packages: django
Successfully installed django-1.10.4

Install Django on Mac

How to upgrade pip on Mac

Execute the following:

sudo pip install --upgrade pip

If the isntall is successful, you should see something like the following:


Collecting pip
Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 517kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.2
Uninstalling pip-8.1.2:
Successfully uninstalled pip-8.1.2
Successfully installed pip-9.0.1

How to upgrade pip on Mac

NumPy Basics and Examples

Let’s start with the basics. Import the library, create a list and then convert it to an array with NumPy.

Screen Shot 2016-03-10 at 10.00.40 PM

Let’s create another list, combine the two and then put both of them to an array. The array at this point becomes two dimensional.
Screen Shot 2016-03-10 at 10.09.25 PM

If you want to know the shape of an array, try the follow
Screen Shot 2016-03-10 at 10.12.38 PM

This returns a tuple and it shows my array is 2 by 4. In other words, my array has 2 rows and 4 columns. You can use the same way to create 3, 4, 5, …. n number of rows and columns. You can also easily initialize your array with zeros. There are two ways:
Screen Shot 2016-03-10 at 10.25.49 PM

Finding the data type of an array is easy as follows:
Screen Shot 2016-03-10 at 10.29.02 PM

To create a 5×5 array of 1s, use the following method:
Screen Shot 2016-03-10 at 10.36.22 PM

To create an Identity Matrix, use the eye method like the following:
Screen Shot 2016-03-10 at 10.39.59 PM

arange is a very cool method you can use to initialize your array. Check the documentation for details but here are some examples:
Screen Shot 2016-03-10 at 10.45.52 PM

You can also initialize by providing what value to start at, when to end and how many spaces/steps between values. Here is what I mean:
Screen Shot 2016-03-10 at 10.51.20 PM

Let’s do some scalar operations on array.
Screen Shot 2016-03-10 at 11.16.25 PM

Let’s reshape an array.
Screen Shot 2016-03-11 at 7.51.52 AM

Transposing an array is easy. Here is how:
Screen Shot 2016-03-11 at 7.53.47 AM

Let’s learn about some Universal Array Functions (in short ufuncs). These are functions you can apply to every value in an array. Details at Python ufunc

Screen Shot 2016-03-11 at 8.14.54 AM

You can also compare the values at each index from the two arrays and return the max.
Screen Shot 2016-03-11 at 8.18.31 AM

We’ll plot some graph and it will require some additional graph plotting library called Matplotlib. I have been using iPython Notebook to show all the code and output here. The next concept will also require some iPython Notebook specific coding to display graphs and I will use comments when needed.

Screen Shot 2016-03-12 at 1.16.08 PM

Screen Shot 2016-03-12 at 1.17.27 PM

Try following the code segments below. Try to experiment with each to have better understanding. We will replace all of it with a shorter version.
Screen Shot 2016-03-12 at 2.16.35 PM

You can replace the above code segments with numpy’s where method like the following:
Screen Shot 2016-03-12 at 2.30.58 PM

Let’s create an 5×5 array with random values.
Screen Shot 2016-03-12 at 2.40.06 PM

We’ll use Numpy’s where method to replace any value in the array that is less zero with zero. We’ll leave everything else the same.
Screen Shot 2016-03-12 at 2.46.11 PM

Let’s learn about some very useful array methods.
Screen Shot 2016-03-12 at 3.04.38 PM
Screen Shot 2016-03-12 at 3.29.15 PM
Screen Shot 2016-03-12 at 3.40.38 PM

Let’s create some sample income data, centered around $54,000 with a normal distribution (remember bell-shaped graph?), a standard deviation of 10,000, and with 10,000 data points. We’ll also look at mean and median values.

Screen Shot 2016-03-12 at 4.02.35 PM

You can save the contents of an array to a file and read it back like the following:
Screen Shot 2016-03-12 at 5.52.50 PM

You can save multiple arrays in a zip file.
Screen Shot 2016-03-12 at 6.01.37 PM

NumPy Basics and Examples

Serving Static File in Django

You need ‘django.contrib.staticfiles’ in our INSTALLED_APPS if you want Django to serve static files.


# Application definition

INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
)

Django will serve the static files at STATIC_URL. By default, STATIC_URL is set to ‘/static/’. It means your static contents like css, js or images are served using the url like http://127.0.0.1:8000/static/styles.css.

So, how does Django know where to read the static files or where to find the static files in your system?

You use STATICFILES_FINDERS property in your settings like the following

STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

Appdirectoriesfinder tells Django to look into static/ subdirectory of each app in INSTALLED_APPS. You would use this approach if you have separate static contents for each installed apps but if you have some common static content that you want to share across multiple apps within your project, you would use STATICFILES_DIRS

STATICFILES_DIRS = (
os.path.join(BASE_DIR, '../common_static'),
)

Add the following at the top of your template file
{% load staticfiles %}

You can reference your static files like the following example in your template files

<link href="{% static 'css/bootstrap.min.css' %}" rel="stylesheet">

Browse to your template file to see the change in effect.

 

 

Serving Static File in Django