Tuesday, December 23, 2008

Can't start new thread

While trying to get some multitasking in python via using vast amount of threads, i've hit my head to "can't start new thread" problem.

At first, i thought that max open file limit was the problem, to verify i did a manual checking as seen below.


In [1]: count = 0
In [2]: from threading import Thread
In [3]: def blah():
...: pass
...:
In [4]: while 1:
...: Thread(target=blah).start()
...: count += 1
...:
...:
---------------------------------------------------------------------------
error Traceback (most recent call last)
/root/ in ()
/usr/local/lib/python2.5/threading.pyc in start(self)
438 _limbo[self] = self
439 _active_limbo_lock.release()
--> 440 _start_new_thread(self.__bootstrap, ())
441 self.__started = True
442 _sleep(0.000001) # 1 usec, to let the thread run (Solaris hack)
error: can't start new thread
In [5]: print count
438

438.. ôO'

i was expecting 1024 or so(mostly more than 1024 if running as root) but 438 was way lower than that. It seemed that another problem was going on.

 root@s1 [~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 65536
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

stack_size(10mb) seems to be the culprit here. stack_size is basically the maximum memory size a single process can allocate as stack.
10Mbs is much more than needed for my threads, so retrying the manual checking succeeded as expected.
 In [1]: import threading as t
In [2]: def blah():
...: pass
...:
In [3]: t.stack_size(128*1024) # 128K would be enough in the real code, so trying it here
Out[3]: 0
In [4]: count = 0
In [5]: while 1:
...: t.Thread(target=blah).start()
...: count +=1
...:
...:
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
.........

KeyboardInterrupt: # bored waiting :o/

In [6]: count
Out[6]: 68348 <------ Yays!

Now i'm happily enjoying my gazillions of semi-concurrent threads.

0 comments:

About Me

hinoglu
Istanbul, Turkey
I'm a self-taught programmer in a developing country, trying to keep my business up and running.

I'll use this space as a sketchboard to improve my writing skills and a notebook for my adventures in business.
View my complete profile