I was dealing with a two queries, knowing:

SELECT oid,typname,typlen,typlem,typdefault,typbasetype,typnotnull,typtype
FROM pg_type;

And

SELECT attname,attnum,atttypid,attndims,attnotnull,atthasdef,
pg_get_expr(adbin,adrelid) AS adbin
FROM pg_attribute LEFT JOIN pg_attrdef ON attrelid = adrelid AND attnum = adnum
WHERE attisdropped = false AND attnum > 0 AND attrelid IN
( SELECT oid FROM pg_class WHERE relname=%s AND relkind='r' ORDER BY attnum);

The code is pretty simple:

import time
t1 = time.time()
      if ' FROM ' in query:
          print 'IN found it!'
      t2 = time.time()
      print "Took me " + str(t2-t1) + " sec."
 
      t3 = time.time()
      if query.find(' FROM ') != -1:
          print 'FIND found it!'
      t4 = time.time()
      print "Took me " + str(t4-t3) + " sec."

The results look as follows:

IN found it!
Took me 4.72068786621e-05 sec.
FIND found it!
Took me 1.09672546387e-05 sec.
IN found it!
Took me 4.19616699219e-05 sec.
FIND found it!
Took me 1.12056732178e-05 sec.
IN found it!
Took me 3.48091125488e-05 sec.
FIND found it!
Took me 9.05990600586e-06 sec.
Took me 1.90734863281e-06 sec.
Took me 5.00679016113e-06 sec.
IN found it!
Took me 2.59876251221e-05 sec.
FIND found it!
Took me 1.19209289551e-05 sec.
Took me 9.53674316406e-07 sec.
Took me 1.90734863281e-06 sec.
IN found it!
Took me 0.00103211402893 sec.
FIND found it!
Took me 2.50339508057e-05 sec.
Took me 9.53674316406e-07 sec.
Took me 4.05311584473e-06 sec.

As we can see: if we use the if-in test, we get only one result even if there are more instances of ' FROM ' in the string. When using the find method, all instances are retrieved. When having only one instance in your string, the find method is usually faster. When having multiple instances, the if-in test will be faster.
It doesn't make much sense with small strings, but if you're just interested in finding a substring one or more times in a large string or a piece of text, it can make a difference.
So far my little experiment. Knowing the answer, I can sleep well again tonight

Yesterday I had an idea about how to manage parsers in dynamic way. My idea was like this

• A managing class can access the parsers folder and check which parsers (files) are available
• A parser describes which file formats (and which versions of it) it can parse

Testfile:

pf = parsers.parserfactory.ParserFactory()
pf.getAvailableParsers()

Then for a single parser I have a GenericParser interface which every parser should implement:

"""
This GenericParser is an abstract superclass. It defines methods which subclasses should override. This guarantees us that
certain functions exist.
@author: Patrick van Kouteren
 
@version: 0.1
"""
 
class GenericParser:
 
    """
    self.fileformats will be a dictionary in the form 'extension : version '. E.g. 'xml : 1.0'
    """
    def __init__(self):
        raise NotImplementedError("This is a GenericParser. Please define a list 'self.fileformat' here with extensions which your parser can parse!")
 
    def getSupportedFileFormats(self):
        raise NotImplementedError("This is a GenericParser. Please return the list 'self.fileformat' here which should be defined at self.init")
 
    def parse(self, filename):
        raise NotImplementedError("This is a GenericParser. Please implement a proper parse function")

The crux is in this getSupportedFileFormats function. I would like to call this function on every parser file. This file contains a parser class, so basically I want to create an object from a file, but I don't know the object's name from the file.

Currently my ParserFactory looks as follows (note that I'm still working on it, so not all is finished yet!):

"""
This ParserFactory contains knowledge about how to parse files. It can be fed a file and return the parsed data.
It uses the parsers in this parsers folder, but abstracts away various operations.
@author: Patrick van Kouteren
 
@version: 0.1
"""
import os, types, sys
 
class ParserFactory:
 
    """
    Gather info about the contents of the database which are important for parsing
    """
    def __init__(self):
        self.importedDatabases = self.checkImportedDatabases()
 
    """
    Check which databases (and which versions of them) are present in the database
    """
    def checkImportedDatabases(self):
        databases = {}
        return databases 
 
    """
    Return a list of databases and their version which are present (imported) in IBIDAS
    """
    def getImportedDatabases(self):
        return self.importedDatabases
 
    """
    Check the parser directory for files which import the GenericParser class. If a file does so, it is guaranteed
    that we can call certain methods to request properties
    """
    def getAvailableParsers(self):
       parserdir = sys.path[0] + "/parsers"
       classnames = []
       parserfiles = []
       for subdir, dirs, files in os.walk(parserdir):
           for file in files:
               if file.endswith(".py") and not file.startswith("__"):
                   parserfiles.append(parserdir + "/" + file)
       for parserfile in parserfiles:
           fileobject = open(parserfile)
           content = fileobject.read()
           importfound = 0
           for l in content.splitlines():
               if l.startswith("import"):
                   """ If this line contains genericparser, we know that this file is interesting """
                   if l.find("genericparser") > 0:
                       importfound += 1
               """
               If we:
                * find a class definition
                * have found an import of generic parser
                * find a genericparser argument
               Then we know that this parser is a subclass of genericparser
               """
               if l.startswith("class") and importfound > 0 and l.find("genericparser") > 0:
                   import re
                   m = re.split("\W+", l)
                   classname = m[m.index("class") + 1]
                   print "file " + parserfile + " contains a callable parser class called " + classname
                   thisfilename = parserfile[parserfile.rfind(u"/")+1:]
                   ppath = "parsers." + thisfilename[:thisfilename.rfind(u".")]+"." + classname
                   #print "the full import path will become " + ppath
                   classnames.append(ppath)
       return classnames
 
    """
    Return the supported file formats. The supported file formats are determined by checking all parsers which import
    the GenericParser. We can request the file formats they support and return this list.
    """
    def getSupportedFileFormats(self):
        fileformats = []
        parserlist = self.getAvailableParsers()
        for parser in parserlist:
            p = self._get_func(parser)()
            for ff in p.getSupportedFileFormats():
                fileformats.append(ff)
        return fileformats
 
    """
    Try to import a module. Then we can use this to get its class
    Source: http://code.activestate.com/recipes/223972/
    """
    def _get_mod(self, modulePath):
        try:
            aMod = sys.modules[modulePath]
            if not isinstance(aMod, types.ModuleType):
                raise KeyError
        except KeyError:
            # The last [''] is very important!
            aMod = __import__(modulePath, globals(), locals(), [''])
            sys.modules[modulePath] = aMod
        return aMod
 
    """
    Return the class from 'parsers.file.class'
    Source: http://code.activestate.com/recipes/223972/
    """
    def _get_func(self,fullFuncName):
        """Retrieve a function object from a full dotted-package name."""
        # Parse out the path, module, and function
        lastDot = fullFuncName.rfind(u".")
        funcName = fullFuncName[lastDot + 1:]
        modPath = fullFuncName[:lastDot]
 
        aMod = self._get_mod(modPath)
        aFunc = getattr(aMod, funcName)
 
        # Assert that the function is a *callable* attribute.
        assert callable(aFunc), u"%s is not callable." % fullFuncName
 
        # Return a reference to the function itself,
        # not the results of the function.
        return aFunc
 
    """
    Returns the class where a method is defined
 
    def find_defining_class(self, obj, meth_name):
        for ty in type(obj).mro():
            if meth_name in ty.__dict__:
                return ty
    """
 
    """
    Check if all databases needed to import a file are present
    """
    def checkPrerequisites(self, file_prerequisites):
        errors = []
 
    """
    Parse a list of files. This means that we not only have to check the prerequisites, but also an order in which to
    parse the files as a file can be a prerequisite for another file
    """
    def parseList(self, filelist, filelist_prerequisites):
        order = self.findParseOrder(file)
 
    def parse(self, file, file_prerequisites, parser=None):
        """ First check if all prerequisites are present """
        errors = self.checkPrerequisites(file_prerequisites)
        if not empty(errors):
            data = "\n".join(errors)
            raise prerequisitesError, data
        else:
            if not parser:
                parser = self.findParser(file)
            else:
                pass
            if parser:
                self.doParsing(file, parser)
            else:
                raise parserError
 
    """
    Based on several things a parser is tried to be found.
    1. The file extension: certain file extensions belong to specific formats
    2. The first line:
    """
    def findParser(self, file):
        pass

Any thoughts, comments and discussions are appreciated. For more information: Chris Leary has posted an improvement here