startup - How to speed up python starting up and/or reduce file search while loading libraries? -

i have framework composed of different tools written in python in multi-user environment.

the first time log system , start 1 command takes 6 seconds show few line of help. if issue same command again takes 0.1s. after couple of minutes gets 6s. (proof of short-lived cache)

the system sits on gpfs disk throughput should ok, though access might low because of amount of files in system.

strace -e open python tool | wc -l 

shows 2154 files being accessed when starting tool.

strace -e open python tool | grep enoent | wc -l 

shows 1945 missing files being looked for. (a bad hit/miss ratio ask me :-)

i have hunch excessive time involved in loading tool consumed querying gpfs files, , these cached next call (at either system or gpfs level), though don't know how test/prove it. have no root access system , can write gpfs , /tmp.

is possible improve python quest missing files?

any idea on how test in simple way? (reinstalling on /tmp not simple, there many packages involved, virtualenv not either (i think), since it's linking files on gpfs system).

an option of course have daemon forks, that's far "simple" , last resort solution.

thanks reading.

how using imp module? in particular there function: imp.find_module(module, path) here http://docs.python.org/2.7/library/imp.html

at least example (see below) reduces number of open() syscalls vs simple 'import numpy,scipy': (update: doesn't possible achieve significant reductions of syscalls way...)

import imp import sys   def loadm(name, path):     fp, pathname, description = imp.find_module(name,[path])     try:         _module = imp.load_module(name, fp, pathname, description)         return _module     finally:         # since may exit via exception, close fp explicitly.         if fp:             fp.close()   numpy = loadm("numpy", "/home/username/py-virtual27/lib/python2.7/site-packages/") scipy = loadm("scipy", "/home/username/py-virtual27/lib/python2.7/site-packages/") 

i guess better check pythonpath empty or small, because can increase loading time.