SQLAlchemy

SQLAlchemy is an open source toolkit for database management mainly though ORM for Python and is release under MIT License. This blog post neither a tutorial nor a self-explanatory sketch of SQLAlchemy. Its merely the info I would need from time to time for my small foolish and dumb applications ;) . I write them so that I can come to them from time to time and I publish them so that if ever anyone needs intersect mine, then great. I’m going to concentrate only on using ORM here.

Connection to the database is done through the engine as shown below:
Engine Structure for SQLAlchemy

Now, since I’m going to use MySQL, I need a dialect tailored for MySQL. The following codes shows how to create an engine:

1
2
from sqlalchemy import create_engine
db = create_engine(URL,arg1..)

where the URL is of the form dialect+driver://user:password@host/dbname[?key=value..]. You can have a look at the list of argument to be passed when creating the engine here. For a list of URLs for specific system, have a look here. Whenever I work, I like to set echo=True just to see the SQL statements being generated.

For MySQL, ensure that you have the library that perform the connection between Python and MySQL. Here is how you can install it:

  1. easy_install mysql-python (mix os)
  2. pip install mysql-python (mix os)
  3. apt-get install python-mysqldb (Linux Ubuntu, …)
  4. cd /usr/ports/databases/py-MySQLdb && make install clean (FreeBSD)
  5. yum install MySQL-python (Linux Fedora, CentOS …)

Now that we have configured the system, we can actually start creating our classes and using the proper construct, all tables gonna generate automatically using the ORM.

1
2
3
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mysql://username:password@localhost/databasename')

Ok, build from the previous code, now suppose that we have define a user like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    username = Column(String(15))
    displayName = Column(String(10))
    password = Column(String(30))
    def __init__(self, username, displayName, password):
        self.username = username
        self.displayName = displayName
        self.password = password
        
    def __repr__(self):
        return "<User('%s','%s', '%s')>" % (self.username, self.displayName, self.password)
Base.metadata.create_all(engine)

The last line, you can comment it from time to time, its this line that automatically creates the tables in the database. Now, lets check in our database. So, we login on mysql and we view all tables show tables;. We view the table user and if we want to see its struture, we just describe user;

Add data to the database

1
2
3
4
5
6
7
username = "noorani"
password = "bakenoor"
displayName = "noor"
emailAddress = "noor@gmail.com"
newUser = User(username, displayName, password,emailAddress)
session = getSession()
session.add(obj)

In fact, several objects can be added in a single command by passing all the objects in a list and then passing this list as an argument to session like session.add_all([newUser,newUser1,newUser2]). Whatever operation done by session stays pending until you finally commit through session.commit().

Querying data from the database

Suppose we want to validate a user with a particular username and password:

1
2
3
username = "noorani"
password = "bakenoor"
user = session.query(db.User.id).filter_by(username=username,password=password).first()

By doing user.username or user.password, you get the username or password respectively. SQLAlchemy has a number of filters:

  • Equals:query.filter(User.name == ‘ed’)
  • Not equals: query.filter(User.name != ‘ed’)
  • Like: query.filter(User.name.like(‘%ed%’))
  • In: query.filter(User.name.in_(session.query(User.name).filter(User.name.like(‘%ed%’))))

As I use SQLAlchemy and gets new stuff, I’m going to post it here

All info to write this post came from:

  1. http://stackoverflow.com/questions/454854/no-module-named-mysqldb
  2. http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html

Broadcom on fedora 17 due to missing firmware

Getting Firmware missing on broadcom fedora 17

broadcom fedora 17.So, I’ve just installed Fedora 17, no great changes in UI, but it’s nice. Now, as always, the problem I get is to set up wireless on laptop(I love my old Dell Inspiron 1545) which uses a broadcom 4312. In case, you are having the same issue, you can follow the procedures below or follow the links found at the end of this post.

First, ensure you have b43-fwcutter,

yum install b43-fwcutter

Now, perform the following:

1 export FIRMWARE_INSTALL_DIR="/lib/firmware"
2 wget http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2
3 tar xjf broadcom-wl-5.100.138.tar.bz2
4 sudo b43-fwcutter -w "$FIRMWARE_INSTALL_DIR" broadcom-wl-5.100.138/linux/wl_apsta.o

The above information has been found in the links below:

1.http://linuxwireless.org/en/users/Drivers/b43#If_you_are_using_the_b43_driver_from_3.2_kernel_or_newer
2.http://ask.fedoraproject.org/question/2053/wireless-broadcom-4312

RIA-Rich Internet Applications

We believe that RIA are modern Internet Application but I think RIA came into existence the moment JavaScipt took birth at Netscape. It’s really at that time that client-side web app started to become “rich”.

Why “Rich” ?
In typical websites, all interface are created on the server-side. The browser only renders the HTML. However, in RIA, server acts only as a model and supplies data. The interface and business logic is implemented in the client code. So, in typical websites, the client is a “thin client” in that it only renders HTML. However, in RIA, as mentioned above, the client is responsible to create the interface and also to implement a connection module to connect to the server.

Wikipedia definition for a RIA consist mainly the following parts:

  • has many of the characteristics of desktop application software
  • typically delivered by way of a site-specific browser, a browser plug-in, an independent sandbox, extensive use of JavaScript, or a virtual machine.”

However, I disagree with this clause “typically delivered by way of a site-specific browser, a browser plug-in”. Several frameworks make use of plugins to display Internet Applications.However, plugin is not a metric to determine whether a web application is an RIA or not. For me, RIAs only mimic desktop application software and try to inherit their traits. Personally, I see 2 main traits that RIAs try to mimic:

1. High Interactivity

In RIA, we don’t wait, they are very responsive and information is on demand(though it is not). In typical websites, users are more tolerant and they consider waiting as natural. However,RIA models desktop applications which tend to be very “fast”. Bandwidth can be a bottleneck leading to latency.Thus,latency need to be hidden/abstracted though it exists. I’ll try to answer the question “How to decrease latency?” below. Also, interaction is not only one way, from users to application but also from application to user. RIA interact with the user by making users to experience their richness through their fantastic designs and behaviors. The design is what I called the Data Presentation. The behaviors are the amazing effects that RIA exhibits such as fadeIn/fadeOut, transitions, drag and drop. In short, RIA need to be “young”, “gorgeous” and must “flirt” with the user in the smoothest way.

2.Data Presentation

On seeing RIAs, you’ll immediately notice that the widgets in the interface is different. Sometimes, you’ll see charts showing updated data on a real-time basis. There’ll be widgets are being updated by xmlHTTPResponses without even the user noticing it. The application structure of a RIA is somewhat different from a normal web application. Consider the following MVP pattern:

Application Structure

If we model an RIA using an MVP pattern then the client-side code is the View and the Presenter. The server only acts as a model for the RIA. All the codes for business logic and interface generation including a module to connect to server makes up the client. Initially all codes are obtained from the server. Thereafter, only data is obtained from the server through request made by the client communication module.

Now, I’m in a better position to answer the question, “How to decrease latency?”
1. Implement the MVP pattern described above to implement your RIA. By doing so, you’ll ensure that after the initial application load, all communication between the client and the server involves only and only bits of data.

2. Caching & Browser local storage
Implement your application such ensure caching on the level of the browser. Also, treat visibility properly so that if ever, you are using HTTP, intermediate proxies can cache your data. Then, for data that can be reused in the future, make use of the local storage.

3. Initial Application load
Normally, RIA are big in size compared to typical websites. So, at the initial application load, not all files are needed. I think it’s best not to load the entire application in one shot because several part might not be of use. Instead, insert junctures in your application that request the needed part from the server when needed. With JavaScript, it’s fairly easy to insert a script in DOM. Also, GWT does this well through a technique which they called Code Splitting

One very important aspect of Human Computer Interface is the enjoyment derived from using the software. So, at the same time, RIA must be at the same time useful and fun. I think that by optimizing bandwidth, ensuring a high level of interactivity and presenting data using rich widgets will allow to develop a good RIA which users will love.

Constaints

Though the technology exists for building RIAs, there are some constrains that cannot be avoided such as:

  • Heterogeneity
  • Search Engine Optimization
  • Browser Features
1.Heterogeneity

Browser Statistics

http://www.w3schools.com/browsers/browsers_stats.asp

Users on the other end make use of several browsers. We all know that browsers have incompatibilities. So, sometimes, we need to be good web developers and at the same time browser gurus to ensure consistency on all available browsers. Sometimes, when implementing some features of RIA, browser sniffing will be important. It’s boring because sometimes a particular feature must be implemented for several browsers if there is not a standard way to do it.

2.Search Engine Optimization

RIA will make use of asynchronous communication to request data from the server. The technology for achieving this is AJAX. There has always been a discussion whether not AJAX is SEO Friendly(Check this discussion here). However, Google has published a technique here to ensure that web crawlers can craw Ajaxified sites. They say it works and I’m sure it does but I’ve never used it.

3.Browser Features

RIA will always exist and so will normal websites do. Normal websites comply with browser features such as browser back/forward button, history, bookmarking. Normally, RIA are ajaxified internet application. Therefore, you’ll need to implement workarounds to ensure you don’t break those functionalities.

Development of RIA

There are several frameworks available for the development of RIAs. However, which one to choose? Personally, I’ve noticed that all RIA frameworks mitigate to maximize at least one of the three traits above. Some of the widely known frameworks are from Adobe technologies such as Flash or Flex, jQuery or Microsoft namely Microsoft Silverlight. (Here is a good list of RIA Frameworks) Check this to get an overview of RIA Statistics.

Future of RIA

I’m almost sure that the use of RIA will intensified in the coming years. Below is a study carried in 2006(obtained here):

Currently more and more companies are moving towards cloud computing(Saas) for their business solutions. Such as for accounting, some may be moving towards openERP or for CRM such as SalesForce. So, companies are now preferring Internet Application over Desktop applications. These Internet Application need to match those Desktop applications in terms of features and interactions. There comes RIAs to their rescue.

MYSQL

To create a user and grant all permission to him:

  1. Log in as root mysql -u root
  2. Create the user and sets the password CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
  3. Grant permission, I’ll be granting all permission GRANT ALL PRIVILEGES ON * . * TO 'username'@'localhost';

Google from your terminal

Foo, sometimes, when you are on terminal, its soo lazy to open google chrome from the dock, I just wrote a foolish script that does it:

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
args=("$@")
all_terms=${args[0]}
i=1
while [ $i -lt ${#args[@]} ]
do
        all_terms=$all_terms+${args[i]}
        i=$[$i+1]
done
google-chrome http://www.google.co.uk/#q=$all_terms

Well, just copy the above script in a file, chmod +x it, all it as an alias using may be “google” as alias name to your .bashrc and source .bashrc and have fun :P

Linux Tricks

ls with full absolute path names:
ls -d $PWD/*

Openshift get data from Data folder

On Openshift PAAS Python, to be able to get data from the Data directory:
file = os.path.join(os.environ["OPENSHIFT_DATA_DIR"],"smallTree.xml")

This is how you can retrive the file from the “data” directory

Create XML from String

Sometimes, we have a String which we just want to serialize as an XML element, so to convert it to XML:

1
2
3
4
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dBuilder.parse(fXmlFile);
Document actualElementXMLResult = dBuilder.parse
(new InputSource(new StringReader("<YourXMLElement>YourXMLElementGoesHere</YourXMLElement>")));

Remember Password, its easy !!

I just had a discussion with a friend, always saying that he forgets his password, but today we don’t have to remember our password. We have tools, very good and free that allows us to do easily remember our password and login. I think for mobile, I think you do have a free trail but for Desktop, the usage it free. Some times ago, I was in the same situation, always trying to remember my password. Then, once I saw Avinash using a very nice tool for password management which was LastPass. From then I used it, and its brilliant, Great :-) . In Summo, this is what LastPass does, well, from what my usage, thats what I know:

  1. Generate strong password for you
  2. Remember password when you enter them on a new site
  3. On entering a site, it allows to choose from a set of credients(i.e. availability )
  4. Automatically set a set of credentials for a site
  5. Synchronize your password with your account

So, why do we need to remember our password when LastPass can do it, it remembers and even enters it for us, well, why do we even pretend to be able to remember our password, sometimes when I see people trying to computer huge calculation by the head, you see its fine, well at least sometimes surprising to see, but why do we need to do things when machines can do them, Wherever we can we automate :)

Your own Github-like System

I know, Git is great and so is Github, fortunately or unfortunately, for private projects, we have to pay. Well, Github has cost and it needs to run and by coincidence money is needed, so it can’t help. No worries, open source is here, there are number of alternatives out there that can help you to host your own Github-like System. Here are some of them:

In case you know some more alternatives, just drop down a comment, I’ll add them to the list

DBpedia SPARQL Examples

DBpedia is really fun, the whole of Wikipedia as an ontology, added it is the DBpedia SPARQL endpoint. Here are some examples I went through while learning. Resources lacks so you to make your way though forums specially Stackoverflow, thats how I did it ;)

Check if a resource exist on Wikipedia

1
2
3
4
SELECT DISTINCT * WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  $z
}

Getting the relationship between London and UK

1
2
3
4
SELECT ?property  
WHERE {
  :London ?property :United_Kingdom
}

Get Country & Cities

1
2
3
4
5
SELECT DISTINCT ?city ?country 
WHERE { ?city rdf:type dbpedia-owl:City ; 
              rdfs:label ?label ; 
              dbpedia-owl:country ?country 
}

Getting the type of a subject and a value that can relate themselves between a predicate country

1
2
3
4
5
SELECT DISTINCT ?w,?z WHERE {
     ?x <http://dbpedia.org/ontology/country> ?y .
     ?x rdf:type  ?z .
     ?x rdf:type ?w
}

Getting the name of type of a resource with name Mauritius

1
2
3
4
5
SELECT DISTINCT ?label WHERE {
     :Mauritius rdf:type ?type .
     ?type rdfs:label ?label .
     FILTER (LANG(?label)='en')
}

=============================================
Getting all things of type kingdom

1
2
3
4
5
PREFIX wo:<http://purl.org/ontology/wo/>
SELECT DISTINCT * 
WHERE {
    ?subject rdf:type wo:Kingdom 
}

As I have fun with SPARQL, I’m going to update this page

What the Hell? I need to back up !!

Ok, I’m currently in a shiit, I’m having to backup my stuffs on my servers and this I need to do it quite frequently. So the issue is:

-Automatically backup frequently(check cron jobs if you want to know)

-Naming Convention to timestamp the folder

So, how do I recognise when a file has been copied? We timestamp the folder using a naming convention say dd.mm.yy.hh.mm.ss(day.month.year.hour.minute.second). We use scp to copy the folder to my server, ensure that your public and private keys are setup so that you can automate it. The key here is the naming convention.

scp -r /path/to/folder server@username.com:/path/to/folder/$(date +\%d.\%m.\%y.\%H.\%M.\%S)

Ok, if you really want to automate the whole stuff:

  • Set up your keys to ensure that authorization can be automated
  • Its good to create an alias of the command so that you have a shortcut whenever you want to do it manually
  • create a crontab and add the frequency