SQLAlchemy

SQLAlchemy is an open source toolkit for database management mainly though ORM for Python and is release under MIT License. This blog post neither a tutorial nor a self-explanatory sketch of SQLAlchemy. Its merely the info I would need from time to time for my small foolish and dumb applications ;) . I write them so that I can come to them from time to time and I publish them so that if ever anyone needs intersect mine, then great. I’m going to concentrate only on using ORM here.

Connection to the database is done through the engine as shown below:
Engine Structure for SQLAlchemy

Now, since I’m going to use MySQL, I need a dialect tailored for MySQL. The following codes shows how to create an engine:

1
2
from sqlalchemy import create_engine
db = create_engine(URL,arg1..)

where the URL is of the form dialect+driver://user:password@host/dbname[?key=value..]. You can have a look at the list of argument to be passed when creating the engine here. For a list of URLs for specific system, have a look here. Whenever I work, I like to set echo=True just to see the SQL statements being generated.

For MySQL, ensure that you have the library that perform the connection between Python and MySQL. Here is how you can install it:

  1. easy_install mysql-python (mix os)
  2. pip install mysql-python (mix os)
  3. apt-get install python-mysqldb (Linux Ubuntu, …)
  4. cd /usr/ports/databases/py-MySQLdb && make install clean (FreeBSD)
  5. yum install MySQL-python (Linux Fedora, CentOS …)

Now that we have configured the system, we can actually start creating our classes and using the proper construct, all tables gonna generate automatically using the ORM.

1
2
3
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mysql://username:password@localhost/databasename')

Ok, build from the previous code, now suppose that we have define a user like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    username = Column(String(15))
    displayName = Column(String(10))
    password = Column(String(30))
    def __init__(self, username, displayName, password):
        self.username = username
        self.displayName = displayName
        self.password = password
        
    def __repr__(self):
        return "<User('%s','%s', '%s')>" % (self.username, self.displayName, self.password)
Base.metadata.create_all(engine)

The last line, you can comment it from time to time, its this line that automatically creates the tables in the database. Now, lets check in our database. So, we login on mysql and we view all tables show tables;. We view the table user and if we want to see its struture, we just describe user;

Add data to the database

1
2
3
4
5
6
7
username = "noorani"
password = "bakenoor"
displayName = "noor"
emailAddress = "noor@gmail.com"
newUser = User(username, displayName, password,emailAddress)
session = getSession()
session.add(obj)

In fact, several objects can be added in a single command by passing all the objects in a list and then passing this list as an argument to session like session.add_all([newUser,newUser1,newUser2]). Whatever operation done by session stays pending until you finally commit through session.commit().

Querying data from the database

Suppose we want to validate a user with a particular username and password:

1
2
3
username = "noorani"
password = "bakenoor"
user = session.query(db.User.id).filter_by(username=username,password=password).first()

By doing user.username or user.password, you get the username or password respectively. SQLAlchemy has a number of filters:

  • Equals:query.filter(User.name == ‘ed’)
  • Not equals: query.filter(User.name != ‘ed’)
  • Like: query.filter(User.name.like(‘%ed%’))
  • In: query.filter(User.name.in_(session.query(User.name).filter(User.name.like(‘%ed%’))))

As I use SQLAlchemy and gets new stuff, I’m going to post it here

All info to write this post came from:

  1. http://stackoverflow.com/questions/454854/no-module-named-mysqldb
  2. http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html

MYSQL

To create a user and grant all permission to him:

  1. Log in as root mysql -u root
  2. Create the user and sets the password CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
  3. Grant permission, I’ll be granting all permission GRANT ALL PRIVILEGES ON * . * TO 'username'@'localhost';

Google from your terminal

Foo, sometimes, when you are on terminal, its soo lazy to open google chrome from the dock, I just wrote a foolish script that does it:

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
args=("$@")
all_terms=${args[0]}
i=1
while [ $i -lt ${#args[@]} ]
do
        all_terms=$all_terms+${args[i]}
        i=$[$i+1]
done
google-chrome http://www.google.co.uk/#q=$all_terms

Well, just copy the above script in a file, chmod +x it, all it as an alias using may be “google” as alias name to your .bashrc and source .bashrc and have fun :P

Linux Tricks

ls with full absolute path names:
ls -d $PWD/*

Openshift get data from Data folder

On Openshift PAAS Python, to be able to get data from the Data directory:
file = os.path.join(os.environ["OPENSHIFT_DATA_DIR"],"smallTree.xml")

This is how you can retrive the file from the “data” directory

Create XML from String

Sometimes, we have a String which we just want to serialize as an XML element, so to convert it to XML:

1
2
3
4
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dBuilder.parse(fXmlFile);
Document actualElementXMLResult = dBuilder.parse
(new InputSource(new StringReader("<YourXMLElement>YourXMLElementGoesHere</YourXMLElement>")));

Remember Password, its easy !!

I just had a discussion with a friend, always saying that he forgets his password, but today we don’t have to remember our password. We have tools, very good and free that allows us to do easily remember our password and login. I think for mobile, I think you do have a free trail but for Desktop, the usage it free. Some times ago, I was in the same situation, always trying to remember my password. Then, once I saw Avinash using a very nice tool for password management which was LastPass. From then I used it, and its brilliant, Great :-) . In Summo, this is what LastPass does, well, from what my usage, thats what I know:

  1. Generate strong password for you
  2. Remember password when you enter them on a new site
  3. On entering a site, it allows to choose from a set of credients(i.e. availability )
  4. Automatically set a set of credentials for a site
  5. Synchronize your password with your account

So, why do we need to remember our password when LastPass can do it, it remembers and even enters it for us, well, why do we even pretend to be able to remember our password, sometimes when I see people trying to computer huge calculation by the head, you see its fine, well at least sometimes surprising to see, but why do we need to do things when machines can do them, Wherever we can we automate :)

Your own Github-like System

I know, Git is great and so is Github, fortunately or unfortunately, for private projects, we have to pay. Well, Github has cost and it needs to run and by coincidence money is needed, so it can’t help. No worries, open source is here, there are number of alternatives out there that can help you to host your own Github-like System. Here are some of them:

In case you know some more alternatives, just drop down a comment, I’ll add them to the list

DBpedia SPARQL Examples

DBpedia is really fun, the whole of Wikipedia as an ontology, added it is the DBpedia SPARQL endpoint. Here are some examples I went through while learning. Resources lacks so you to make your way though forums specially Stackoverflow, thats how I did it ;)

Check if a resource exist on Wikipedia

1
2
3
4
SELECT DISTINCT * WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  $z
}

Getting the relationship between London and UK

1
2
3
4
SELECT ?property  
WHERE {
  :London ?property :United_Kingdom
}

Get Country & Cities

1
2
3
4
5
SELECT DISTINCT ?city ?country 
WHERE { ?city rdf:type dbpedia-owl:City ; 
              rdfs:label ?label ; 
              dbpedia-owl:country ?country 
}

Getting the type of a subject and a value that can relate themselves between a predicate country

1
2
3
4
5
SELECT DISTINCT ?w,?z WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  ?z .
     ?x rdf:type ?w
}

Getting the name of type of a resource with name Mauritius

1
2
3
4
5
SELECT DISTINCT ?label WHERE {
     :Mauritius rdf:type ?type .
     ?type rdfs:label ?label .
     FILTER (LANG(?label)='en')
}

=============================================
Getting all things of type kingdom

1
2
3
4
5
PREFIX wo:<http://purl.org/ontology/wo/>
SELECT DISTINCT * 
WHERE {
    ?subject rdf:type wo:Kingdom 
}

As I have fun with SPARQL, I’m going to update this page

What the Hell? I need to back up !!

Ok, I’m currently in a shiit, I’m having to backup my stuffs on my servers and this I need to do it quite frequently. So the issue is:

-Automatically backup frequently(check cron jobs if you want to know)

-Naming Convention to timestamp the folder

So, how do I recognise when a file has been copied? We timestamp the folder using a naming convention say dd.mm.yy.hh.mm.ss(day.month.year.hour.minute.second). We use scp to copy the folder to my server, ensure that your public and private keys are setup so that you can automate it. The key here is the naming convention.

scp -r /path/to/folder server@username.com:/path/to/folder/$(date +\%d.\%m.\%y.\%H.\%M.\%S)

Ok, if you really want to automate the whole stuff:

  • Set up your keys to ensure that authorization can be automated
  • Its good to create an alias of the command so that you have a shortcut whenever you want to do it manually
  • create a crontab and add the frequency