I was dealing with a two queries, knowing:

SELECT oid,typname,typlen,typlem,typdefault,typbasetype,typnotnull,typtype
FROM pg_type;

And

SELECT attname,attnum,atttypid,attndims,attnotnull,atthasdef,
pg_get_expr(adbin,adrelid) AS adbin
FROM pg_attribute LEFT JOIN pg_attrdef ON attrelid = adrelid AND attnum = adnum
WHERE attisdropped = false AND attnum > 0 AND attrelid IN
( SELECT oid FROM pg_class WHERE relname=%s AND relkind='r' ORDER BY attnum);

The code is pretty simple:

import time
t1 = time.time()
      if ' FROM ' in query:
          print 'IN found it!'
      t2 = time.time()
      print "Took me " + str(t2-t1) + " sec."
 
      t3 = time.time()
      if query.find(' FROM ') != -1:
          print 'FIND found it!'
      t4 = time.time()
      print "Took me " + str(t4-t3) + " sec."

The results look as follows:

IN found it!
Took me 4.72068786621e-05 sec.
FIND found it!
Took me 1.09672546387e-05 sec.
IN found it!
Took me 4.19616699219e-05 sec.
FIND found it!
Took me 1.12056732178e-05 sec.
IN found it!
Took me 3.48091125488e-05 sec.
FIND found it!
Took me 9.05990600586e-06 sec.
Took me 1.90734863281e-06 sec.
Took me 5.00679016113e-06 sec.
IN found it!
Took me 2.59876251221e-05 sec.
FIND found it!
Took me 1.19209289551e-05 sec.
Took me 9.53674316406e-07 sec.
Took me 1.90734863281e-06 sec.
IN found it!
Took me 0.00103211402893 sec.
FIND found it!
Took me 2.50339508057e-05 sec.
Took me 9.53674316406e-07 sec.
Took me 4.05311584473e-06 sec.

As we can see: if we use the if-in test, we get only one result even if there are more instances of ' FROM ' in the string. When using the find method, all instances are retrieved. When having only one instance in your string, the find method is usually faster. When having multiple instances, the if-in test will be faster.
It doesn't make much sense with small strings, but if you're just interested in finding a substring one or more times in a large string or a piece of text, it can make a difference.
So far my little experiment. Knowing the answer, I can sleep well again tonight

If you don't already have IPython you can install it by opening a console and typing

sudo easy_install ipython

Mac OS X does include a readline functionality, but not 'the real one' gnureadline, because of a license issue. It can be manually installed, which we will do next. If you've not done so already, open a console and type

sudo easy_install -f http://ipython.scipy.org/dist/ readline

Type in your password and the readline functionality will be installed. Big joy you would think, but it's very well possible that it still doesn't work. Try for yourself by typing the following in your console:

ipython

IPython will start. Let's see if we can auto-complete here. First type

import sys

Now we've done an import and we will try to have auto-complete hint what we can do with it. In the next line type

sys.

Press Tab directly after the dot. If IPython hints all types of functions preceded by sys. in a list, you're done with this tutorial. If your cursor just jumps, auto-complete doesn't work and you might want to execute the following steps.
IPython has a file which is used to start it. We need to edit this file to tell where the readline module is installed.
If you're still in IPython, close it by pressing CTRL+D and confirming (y). You are now back on the console again.
The file which we are about to edit (done with Vim here) is a read-only file, so we need to sudo once again. Type the following

sudo vim /usr/local/bin/ipython

Enter your password and Vim will open a config file with just a few lines like

#!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
# EASY-INSTALL-ENTRY-SCRIPT: 'ipython==0.9.1','console_scripts','ipython'
__requires__ = 'ipython==0.9.1'
import sys
from pkg_resources import load_entry_point
 
sys.exit(
   load_entry_point('ipython==0.9.1', 'console_scripts', 'ipython')()
)

First find the location of the egg directory of readline which you've just installed. It's probably something like /Library/Python/site-packages/readline-.egg
I've installed the i386 readline module version 2.5.1, so my location is /Library/Python/2.5/site-packages/readline-2.5.1-py2.5-macosx-10.5-i386.egg
We will need this location next.
Go back to your console where the file is still opened.
Leave the first two lines and the import sys line. The rest can be commented out.
Now insert the following three lines after import sys:

sys.path.insert(0, 'path to readline egg')
import IPython.Shell
IPython.Shell.start().mainloop()

In my case, the ipython file looks as follows:

#!/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
# EASY-INSTALL-ENTRY-SCRIPT: 'ipython==0.9.1','console_scripts','ipython'
#__requires__ = 'ipython==0.9.1'
import sys
#from pkg_resources import load_entry_point
 
sys.path.insert(0, '/Library/Python/2.5/site-packages/readline-2.5.1-py2.5-macosx-10.5-i386.egg')
 
import IPython.Shell
IPython.Shell.start().mainloop()
 
#sys.exit(
#   load_entry_point('ipython==0.9.1', 'console_scripts', 'ipython')()
#)

Save and quit the file by typing

:wq

You're back at the command line again. Let's test the auto-complete. Again type

ipython

IPython will start. Let's see if we can auto-complete here. First type

import sys

Now we've done an import and we will try to have auto-complete hint what we can do with it. In the next line type

sys.

Press Tab directly after the dot. If all went well Python hints all types of functions preceded by sys. in a list.

Sites that helped me and / or might be useful for you:

• IPython on Mac OS X
• Incredible Vehicle
• Launchpad

Yesterday I had an idea about how to manage parsers in dynamic way. My idea was like this

• A managing class can access the parsers folder and check which parsers (files) are available
• A parser describes which file formats (and which versions of it) it can parse

Testfile:

pf = parsers.parserfactory.ParserFactory()
pf.getAvailableParsers()

Then for a single parser I have a GenericParser interface which every parser should implement:

"""
This GenericParser is an abstract superclass. It defines methods which subclasses should override. This guarantees us that
certain functions exist.
@author: Patrick van Kouteren
 
@version: 0.1
"""
 
class GenericParser:
 
    """
    self.fileformats will be a dictionary in the form 'extension : version '. E.g. 'xml : 1.0'
    """
    def __init__(self):
        raise NotImplementedError("This is a GenericParser. Please define a list 'self.fileformat' here with extensions which your parser can parse!")
 
    def getSupportedFileFormats(self):
        raise NotImplementedError("This is a GenericParser. Please return the list 'self.fileformat' here which should be defined at self.init")
 
    def parse(self, filename):
        raise NotImplementedError("This is a GenericParser. Please implement a proper parse function")

The crux is in this getSupportedFileFormats function. I would like to call this function on every parser file. This file contains a parser class, so basically I want to create an object from a file, but I don't know the object's name from the file.

Currently my ParserFactory looks as follows (note that I'm still working on it, so not all is finished yet!):

"""
This ParserFactory contains knowledge about how to parse files. It can be fed a file and return the parsed data.
It uses the parsers in this parsers folder, but abstracts away various operations.
@author: Patrick van Kouteren
 
@version: 0.1
"""
import os, types, sys
 
class ParserFactory:
 
    """
    Gather info about the contents of the database which are important for parsing
    """
    def __init__(self):
        self.importedDatabases = self.checkImportedDatabases()
 
    """
    Check which databases (and which versions of them) are present in the database
    """
    def checkImportedDatabases(self):
        databases = {}
        return databases 
 
    """
    Return a list of databases and their version which are present (imported) in IBIDAS
    """
    def getImportedDatabases(self):
        return self.importedDatabases
 
    """
    Check the parser directory for files which import the GenericParser class. If a file does so, it is guaranteed
    that we can call certain methods to request properties
    """
    def getAvailableParsers(self):
       parserdir = sys.path[0] + "/parsers"
       classnames = []
       parserfiles = []
       for subdir, dirs, files in os.walk(parserdir):
           for file in files:
               if file.endswith(".py") and not file.startswith("__"):
                   parserfiles.append(parserdir + "/" + file)
       for parserfile in parserfiles:
           fileobject = open(parserfile)
           content = fileobject.read()
           importfound = 0
           for l in content.splitlines():
               if l.startswith("import"):
                   """ If this line contains genericparser, we know that this file is interesting """
                   if l.find("genericparser") > 0:
                       importfound += 1
               """
               If we:
                * find a class definition
                * have found an import of generic parser
                * find a genericparser argument
               Then we know that this parser is a subclass of genericparser
               """
               if l.startswith("class") and importfound > 0 and l.find("genericparser") > 0:
                   import re
                   m = re.split("\W+", l)
                   classname = m[m.index("class") + 1]
                   print "file " + parserfile + " contains a callable parser class called " + classname
                   thisfilename = parserfile[parserfile.rfind(u"/")+1:]
                   ppath = "parsers." + thisfilename[:thisfilename.rfind(u".")]+"." + classname
                   #print "the full import path will become " + ppath
                   classnames.append(ppath)
       return classnames
 
    """
    Return the supported file formats. The supported file formats are determined by checking all parsers which import
    the GenericParser. We can request the file formats they support and return this list.
    """
    def getSupportedFileFormats(self):
        fileformats = []
        parserlist = self.getAvailableParsers()
        for parser in parserlist:
            p = self._get_func(parser)()
            for ff in p.getSupportedFileFormats():
                fileformats.append(ff)
        return fileformats
 
    """
    Try to import a module. Then we can use this to get its class
    Source: http://code.activestate.com/recipes/223972/
    """
    def _get_mod(self, modulePath):
        try:
            aMod = sys.modules[modulePath]
            if not isinstance(aMod, types.ModuleType):
                raise KeyError
        except KeyError:
            # The last [''] is very important!
            aMod = __import__(modulePath, globals(), locals(), [''])
            sys.modules[modulePath] = aMod
        return aMod
 
    """
    Return the class from 'parsers.file.class'
    Source: http://code.activestate.com/recipes/223972/
    """
    def _get_func(self,fullFuncName):
        """Retrieve a function object from a full dotted-package name."""
        # Parse out the path, module, and function
        lastDot = fullFuncName.rfind(u".")
        funcName = fullFuncName[lastDot + 1:]
        modPath = fullFuncName[:lastDot]
 
        aMod = self._get_mod(modPath)
        aFunc = getattr(aMod, funcName)
 
        # Assert that the function is a *callable* attribute.
        assert callable(aFunc), u"%s is not callable." % fullFuncName
 
        # Return a reference to the function itself,
        # not the results of the function.
        return aFunc
 
    """
    Returns the class where a method is defined
 
    def find_defining_class(self, obj, meth_name):
        for ty in type(obj).mro():
            if meth_name in ty.__dict__:
                return ty
    """
 
    """
    Check if all databases needed to import a file are present
    """
    def checkPrerequisites(self, file_prerequisites):
        errors = []
 
    """
    Parse a list of files. This means that we not only have to check the prerequisites, but also an order in which to
    parse the files as a file can be a prerequisite for another file
    """
    def parseList(self, filelist, filelist_prerequisites):
        order = self.findParseOrder(file)
 
    def parse(self, file, file_prerequisites, parser=None):
        """ First check if all prerequisites are present """
        errors = self.checkPrerequisites(file_prerequisites)
        if not empty(errors):
            data = "\n".join(errors)
            raise prerequisitesError, data
        else:
            if not parser:
                parser = self.findParser(file)
            else:
                pass
            if parser:
                self.doParsing(file, parser)
            else:
                raise parserError
 
    """
    Based on several things a parser is tried to be found.
    1. The file extension: certain file extensions belong to specific formats
    2. The first line:
    """
    def findParser(self, file):
        pass

Any thoughts, comments and discussions are appreciated. For more information: Chris Leary has posted an improvement here