SQLAlchemy

SQLAlchemy is an open source toolkit for database management mainly though ORM for Python and is release under MIT License. This blog post neither a tutorial nor a self-explanatory sketch of SQLAlchemy. Its merely the info I would need from time to time for my small foolish and dumb applications ;) . I write them so that I can come to them from time to time and I publish them so that if ever anyone needs intersect mine, then great. I’m going to concentrate only on using ORM here.

Connection to the database is done through the engine as shown below:
Engine Structure for SQLAlchemy

Now, since I’m going to use MySQL, I need a dialect tailored for MySQL. The following codes shows how to create an engine:

1
2
from sqlalchemy import create_engine
db = create_engine(URL,arg1..)

where the URL is of the form dialect+driver://user:password@host/dbname[?key=value..]. You can have a look at the list of argument to be passed when creating the engine here. For a list of URLs for specific system, have a look here. Whenever I work, I like to set echo=True just to see the SQL statements being generated.

For MySQL, ensure that you have the library that perform the connection between Python and MySQL. Here is how you can install it:

  1. easy_install mysql-python (mix os)
  2. pip install mysql-python (mix os)
  3. apt-get install python-mysqldb (Linux Ubuntu, …)
  4. cd /usr/ports/databases/py-MySQLdb && make install clean (FreeBSD)
  5. yum install MySQL-python (Linux Fedora, CentOS …)

Now that we have configured the system, we can actually start creating our classes and using the proper construct, all tables gonna generate automatically using the ORM.

1
2
3
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mysql://username:password@localhost/databasename')

Ok, build from the previous code, now suppose that we have define a user like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    username = Column(String(15))
    displayName = Column(String(10))
    password = Column(String(30))
    def __init__(self, username, displayName, password):
        self.username = username
        self.displayName = displayName
        self.password = password
        
    def __repr__(self):
        return "<User('%s','%s', '%s')>" % (self.username, self.displayName, self.password)
Base.metadata.create_all(engine)

The last line, you can comment it from time to time, its this line that automatically creates the tables in the database. Now, lets check in our database. So, we login on mysql and we view all tables show tables;. We view the table user and if we want to see its struture, we just describe user;

Add data to the database

1
2
3
4
5
6
7
username = "noorani"
password = "bakenoor"
displayName = "noor"
emailAddress = "noor@gmail.com"
newUser = User(username, displayName, password,emailAddress)
session = getSession()
session.add(obj)

In fact, several objects can be added in a single command by passing all the objects in a list and then passing this list as an argument to session like session.add_all([newUser,newUser1,newUser2]). Whatever operation done by session stays pending until you finally commit through session.commit().

Querying data from the database

Suppose we want to validate a user with a particular username and password:

1
2
3
username = "noorani"
password = "bakenoor"
user = session.query(db.User.id).filter_by(username=username,password=password).first()

By doing user.username or user.password, you get the username or password respectively. SQLAlchemy has a number of filters:

  • Equals:query.filter(User.name == ‘ed’)
  • Not equals: query.filter(User.name != ‘ed’)
  • Like: query.filter(User.name.like(‘%ed%’))
  • In: query.filter(User.name.in_(session.query(User.name).filter(User.name.like(‘%ed%’))))

As I use SQLAlchemy and gets new stuff, I’m going to post it here

All info to write this post came from:

  1. http://stackoverflow.com/questions/454854/no-module-named-mysqldb
  2. http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html

Dynamically create Html using JavaScript

Don’t wrap HTML in JavaScript code, Keep them seperate else they’ll fight and you’ll get beaten

Today, the trend is “Mashups”, we take data from this web service and that web service and make something beautiful. We normally do this in javaScript, making xmlHTTPRequest. On obtaining a response. we then dynamically use JavaScript to create HTML. However, I’ve notice that many developers tend to mix JavaScript and HTML. By doing so, I think they complicate the codes. Representing HTML codes as JavaScript Strings make them prone to JavaScript errors. Also, they might end in making their HTML not as per W3C recommendation due to single and double quotes.

Suppose I’m getting the details of laptops from an xmlHTTPRequest. Then, in JavaScript, I’m having to generate a layout to display the laptop’s details.

I get the laptop brand name, screen size, processor capacity, RAM , Hard Disk details then a picture URL for the laptop and i need to write JavaScript to build an HTML layout and display the content in it.

Codes to dynamically use JavaScript to create HTML

1. The bad practice, Mixing the codes(this is what i think)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$.get("URL_to_get_details",function(laptops)
{
      /*here i need to write the code to display the laptop
      suppose the laptop is json encoded */
      laptops=$.parseJSON(laptops); 
      /*parse the json encoded data with jQuery*/
      var i=0,html="";
      while (i<laptops.length){
           currentLaptop=laptops[i];
           /*build the HTML for the laptops*/
           html+="<div><span>Brand Name:"+currentLaptop.brandName+"</span><span>"+.....+
                 "<img src= "\"+currentlaptop.URL+"\" alt=\"create HTML using JavaScript\">";
      }
      /* create the newly built layout */
      ("#laptopContainer").html(html);
});

What I’m doing in the above code is to wrap HTML in JavaScript. I’m representing HTML as a JavaScript String. I’m having to be cautious due to JavaScript String quotes and HTML quotes. I really don’t think its good. When you start to have lots of HTML, you’ll become crazy mingling with HTML syntax and keeping JavaScript free from errors at the same time.

So this is what i propose. Completly isolate HTML and JavaScript codes. Instead of wrapping HTML into JavaScript, declare a prototype of the layout in HTML and hide it in your DOM.

2. The good practice, Seperating the codes

Normally, I do it like this because i think it’s better. If you have a better solution please share it :)

To Use JavaScript to create HTML:

  • Declare an HTML Layout prototype in the DOM with dummy variables
  • Use raw JavaScript or jQuery to retrieve the HTML codes from DOM as string
  • Obtain variable from their sources
  • use JavaScript to replace dummy variable with apporiate values(e.g use .replace)
1
2
3
4
5
6
7
8
9
10
<div id="prototype" style="display:none">
    <div>
       <span>Brand Name: varBrandName</span>
       <span>Screen Size: varScreenSize</span>
       <span>Processor Capacity: varProcessorCapacity</span>
       <span>RAM:varRAM</span>
       <span>Hard Disk Details: varHardDisk</span>
       <span><img src="varPictureSrc" alt="img" /></span> 
   <div>
</div>

now, in the HTML code above, you’ll see var. These will be actually be replaced by the actual values in JavaScript.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$.get("URL_to_get_details",function(laptops)
{
      /*here i need to write the code to display the laptop
      suppose the laptop is json encoded */
      laptops=$.parseJSON(laptops); /*parse the json encoded data with jQuery*/
      var i=0,html="";
      while (i<laptops.length){
           currentLaptop=laptops[i];
         
           /* get the content of the prototype */
           currentHTML=("#prototype").html();
   
           /* replace the dummy values in the HTML */
           currentHTML=currentHTML.replace("varBrandName",laptops[i].brandName);
           currentHTML=currentHTML.replace("varScreenSize",laptops[i].screenSize)
           currentHTML=currentHTML.replace("varProcessorCapacity",laptops[i].processor);
           currentHTML=currentHTML.replace("varRAM",laptops[i].RAM);
           currentHTML=currentHTML.replace("varHardDisk",laptops[i].hardDisk);
           currentHTML=currentHTML.replace("varPictureSrc",laptops[i].pictureSrc);
     
           /* build the HTML for the laptops */
           html+=currentHTML;
      }
      /* create the newly built layout */
      ("#laptopContainer").html(html);
});

SO, as you see, I’ve retrive the HTML from the declared prototype and then generated the layout replacing the dummy variables with the value I’ve obtained from xmlHTTPResponse. At any time, I did not wrap the HTML in JavaScript. The prototype I declared is pure HTML as per W3C recommendation.

Facebook offline_access is offline now

Okk, “moment de verite”, Facebook offline_access extended permission has been on 5th December 2012. Why? I don’t know but if I ask myself, I think its mainly because of security or abusive use.

So, what happens to those who are currently using Facebook offline_access extended permission?

<h2>How is it now?</h2>
Though there is no longer offline access, there is the capability to extend short lived session. Facebook automatically renews short lived session I think every 1 to 2 hours but if you want to use the access token for longer period, ensure that you renew it. So, yeah, for now, to confirm, long-lived session are for 60 days while short-lived one are for 1-2 hours.

for those who were not using the extended permission offline_access, you have no changes to make to your application. But application using offline_access will have to add some chunks of code for new users. If you were using offline_access extended permission, this will no longer be asked in the application login dialog box when the user uses your application. For new users, you’ll get a short-lived token which you will use to extend to get a long-lived one. Now, if you try to extend a token which already a long-lived one, you may get the same token or a token with a longer expiration time.

To get a long-lived session, just make a call on this URL with the required parameters replaced,

https://graph.facebook.com/oauth/access_token?

client_id=APP_ID&
client_secret=APP_SECRET&
grant_type=fb_exchange_token&
fb_exchange_token=EXISTING_ACCESS_TOKEN

So, it make sense for you to have a short-lived token before submitting the above request

Further information goes here:)
1. http://developers.facebook.com/roadmap/offline-access-removal/
2. http://developers.facebook.com/docs/howtos/login/extending-tokens/
3. http://stackoverflow.com/questions/10487049/facebook-long-lived-and-short-lived-access-tokens-and-their-expirancy-after-off

MYSQL

To create a user and grant all permission to him:

  1. Log in as root mysql -u root
  2. Create the user and sets the password CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
  3. Grant permission, I’ll be granting all permission GRANT ALL PRIVILEGES ON * . * TO 'username'@'localhost';

Google from your terminal

Foo, sometimes, when you are on terminal, its soo lazy to open google chrome from the dock, I just wrote a foolish script that does it:

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
args=("$@")
all_terms=${args[0]}
i=1
while [ $i -lt ${#args[@]} ]
do
        all_terms=$all_terms+${args[i]}
        i=$[$i+1]
done
google-chrome http://www.google.co.uk/#q=$all_terms

Well, just copy the above script in a file, chmod +x it, all it as an alias using may be “google” as alias name to your .bashrc and source .bashrc and have fun :P

Linux Tricks

ls with full absolute path names:
ls -d $PWD/*

Openshift get data from Data folder

On Openshift PAAS Python, to be able to get data from the Data directory:
file = os.path.join(os.environ["OPENSHIFT_DATA_DIR"],"smallTree.xml")

This is how you can retrive the file from the “data” directory

Create XML from String

Sometimes, we have a String which we just want to serialize as an XML element, so to convert it to XML:

1
2
3
4
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dBuilder.parse(fXmlFile);
Document actualElementXMLResult = dBuilder.parse
(new InputSource(new StringReader("<YourXMLElement>YourXMLElementGoesHere</YourXMLElement>")));

Remember Password, its easy !!

I just had a discussion with a friend, always saying that he forgets his password, but today we don’t have to remember our password. We have tools, very good and free that allows us to do easily remember our password and login. I think for mobile, I think you do have a free trail but for Desktop, the usage it free. Some times ago, I was in the same situation, always trying to remember my password. Then, once I saw Avinash using a very nice tool for password management which was LastPass. From then I used it, and its brilliant, Great :-) . In Summo, this is what LastPass does, well, from what my usage, thats what I know:

  1. Generate strong password for you
  2. Remember password when you enter them on a new site
  3. On entering a site, it allows to choose from a set of credients(i.e. availability )
  4. Automatically set a set of credentials for a site
  5. Synchronize your password with your account

So, why do we need to remember our password when LastPass can do it, it remembers and even enters it for us, well, why do we even pretend to be able to remember our password, sometimes when I see people trying to computer huge calculation by the head, you see its fine, well at least sometimes surprising to see, but why do we need to do things when machines can do them, Wherever we can we automate :)

Your own Github-like System

I know, Git is great and so is Github, fortunately or unfortunately, for private projects, we have to pay. Well, Github has cost and it needs to run and by coincidence money is needed, so it can’t help. No worries, open source is here, there are number of alternatives out there that can help you to host your own Github-like System. Here are some of them:

In case you know some more alternatives, just drop down a comment, I’ll add them to the list

DBpedia SPARQL Examples

DBpedia is really fun, the whole of Wikipedia as an ontology, added it is the DBpedia SPARQL endpoint. Here are some examples I went through while learning. Resources lacks so you to make your way though forums specially Stackoverflow, thats how I did it ;)

Check if a resource exist on Wikipedia

1
2
3
4
SELECT DISTINCT * WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  $z
}

Getting the relationship between London and UK

1
2
3
4
SELECT ?property  
WHERE {
  :London ?property :United_Kingdom
}

Get Country & Cities

1
2
3
4
5
SELECT DISTINCT ?city ?country 
WHERE { ?city rdf:type dbpedia-owl:City ; 
              rdfs:label ?label ; 
              dbpedia-owl:country ?country 
}

Getting the type of a subject and a value that can relate themselves between a predicate country

1
2
3
4
5
SELECT DISTINCT ?w,?z WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  ?z .
     ?x rdf:type ?w
}

Getting the name of type of a resource with name Mauritius

1
2
3
4
5
SELECT DISTINCT ?label WHERE {
     :Mauritius rdf:type ?type .
     ?type rdfs:label ?label .
     FILTER (LANG(?label)='en')
}

=============================================
Getting all things of type kingdom

1
2
3
4
5
PREFIX wo:<http://purl.org/ontology/wo/>
SELECT DISTINCT * 
WHERE {
    ?subject rdf:type wo:Kingdom 
}

As I have fun with SPARQL, I’m going to update this page

What the Hell? I need to back up !!

Ok, I’m currently in a shiit, I’m having to backup my stuffs on my servers and this I need to do it quite frequently. So the issue is:

-Automatically backup frequently(check cron jobs if you want to know)

-Naming Convention to timestamp the folder

So, how do I recognise when a file has been copied? We timestamp the folder using a naming convention say dd.mm.yy.hh.mm.ss(day.month.year.hour.minute.second). We use scp to copy the folder to my server, ensure that your public and private keys are setup so that you can automate it. The key here is the naming convention.

scp -r /path/to/folder server@username.com:/path/to/folder/$(date +\%d.\%m.\%y.\%H.\%M.\%S)

Ok, if you really want to automate the whole stuff:

  • Set up your keys to ensure that authorization can be automated
  • Its good to create an alias of the command so that you have a shortcut whenever you want to do it manually
  • create a crontab and add the frequency