damnsweet

Sunday, August 23, 2009

Code completion in twisted manhole shell

It's been short since i've found out that it's possible to digg a hole through twisted applications via manhole. Trying it out, seemed so beautiful, but an instant problem bugged me that there's no support for code completion, which is essential for people with weak memories, like me.

Here's a dirty solution for code completion in manhole shell:


from twisted.cred import portal, checkers
from twisted.conch import manhole, manhole_ssh
from rlcompleter import Completer

class EnhancedColoredManhole(manhole.ColoredManhole):

def find_common(self, l):
''' find common parts in thelist items
ex: 'ab' for ['abcd','abce','abf']
requires an ordered list
'''
if len(l) == 1:
return l[0]

init = l[0]
for item in l[1:]:
for i, (x,y) in enumerate(zip(init, item)):
if x != y:
init = "".join(init[:i])
break

if not init:
return None
return init

def handle_TAB(self):
necessarypart = "".join(self.lineBuffer).split(' ')[-1]
completer = Completer(globals())
if completer.complete(necessarypart, 0):
matches = list(set(completer.matches)) # has multiples

if len(matches) == 1:
length = len(necessarypart)
self.lineBuffer = self.lineBuffer[:-length]
self.lineBuffer.extend(matches[0])
self.lineBufferIndex = len(self.lineBuffer)
else:
matches.sort()
commons = self.find_common(matches)
if commons:
length = len(necessarypart)
self.lineBuffer = self.lineBuffer[:-length]
self.lineBuffer.extend(commons)
self.lineBufferIndex = len(self.lineBuffer)

self.terminal.nextLine()
while matches:
matches, part = matches[4:], matches[:4]
for item in part:
self.terminal.write('%s' % item.ljust(30))
self.terminal.write('\n')
self.terminal.nextLine()

self.terminal.eraseLine()
self.terminal.cursorBackward(self.lineBufferIndex + 5)
self.terminal.write("%s %s" % (self.ps[self.pn], "".join(self.lineBuffer)))


def keystrokeReceived(self, keyID, modifier):
self.keyHandlers.update({'\b':self.handle_BACKSPACE}) # my terminal needed this
m = self.keyHandlers.get(keyID)
if m is not None:
m()
elif keyID in string.printable:
self.characterReceived(keyID, False)


def getManholeFactory(namespace):
realm = manhole_ssh.TerminalRealm()
def getManhole(_):
return EnhancedColoredManhole(namespace)
realm.chainedProtocolFactory.protocolFactory = getManhole
p = portal.Portal(realm)
p.registerChecker(
checkers.InMemoryUsernamePasswordDatabaseDontUse(admin='foobar'))
f = manhole_ssh.ConchFactory(p)
return f

reactor.listenTCP(2222, getManholeFactory(globals()))

Tuesday, February 10, 2009

Lesson 2: Keep it workin'

This article is somewhat a draft, and will be modified constantly till it's much better than this version


Today i had the chance of having a meeting with two senior ladies over a possible website job.

It was a plain old style meeting. Technical guy (me) introduces himself, talks about how superly awesome he is and how he'll handle the tecchi stuff easily and meet the requirements needed for the job blah and blah...

Well.. was not much like that. Indeed i introduced myself, but rest of the story was slightly different.

I was not the only tecchi person there. Two senior ladies did a wonderful job having their time studying web technologies, current implementations and stuff. They were asking me about Verizon, ssl keys, rss feeds, membership isues, advertisement revenues and stuff. They've even shown me the books they studied. Many notes were taken, pages were bookmarked. Wow..

Those ladies were the executives of one of the biggest publishers in the country, maybe Europe too.

Lesson learned is to keep the damn thing working. No matter what your age is, in whatever situation you are or whatever you are doing. Just keep studying and learning. Success will eventually follow.

Saturday, January 17, 2009

Lesson 1: Keep it simple

Lately i had the chance to present some of my project ideas to the vp of strategy and business development dept. of a big telecommunications company.

I love these meetings. Feeling the information and experience floating around waiting to be sucked dry is so aawsumm (information fanboy here!)..

Having only technical backgrounds, I do not have much experience in presentations, nor do i have above-the-average communications skills.

But i have the general idea that presentations should be done using powerpoint, should provide a general information, some statistical information, some plots, nice colors and stuff.. those would do, right?.. Right?

Well, back in the 90ies those were used for the entertainment purposes for the managers from the jurassic ages. Shiny colors, letters dancing on the screen, objects blinking with happiness.. God how those old bags really loved them.

"Just cut the crap already and tell me what do you want" was what i got when i began throwing percentages and statistics around.

He instantly taught me a lesson there. I was there for a reason, and any reason can be briefed to few simple sentences. All i needed to do was just put all the bullcrap aside and tell them "what" i wanted. We all are busy people, and we all are in need of accomplishing something, so why so formal? Just get to damn point already.

Thursday, December 25, 2008

Might & Magic

"Twiki is a flexible, powerful, and easy to use enterprise wiki, enterprise collaboration platform and knowledge management system." as directly pinched from the project's site.

We tend to believe that "powerful" mostly means "mighty" as "it has some overhuman capabilites that makes life unbelievable easy for us, that all we have to do is sit back and watch the process while sipping our delicious fruit coctail or whatsoever...".

Does "powerful explosive" fit to this description?
I guess not. In general cases, "destructive" is much more suitable.

What if we narrow the context from "general" to "mining" in the example above? Now we are talking about a useful tool which might hand one's ass to them in pieces when used without care.

Context is important when defining problems and solutions.

Twiki is powerful when it comes simple everyday operations that one can expect from a wiki for document management. When more (where controls and programing take place) is desired, it turns out that the mighty tool is indeed a powerful one that should be handled carefully. If not, then the days of hell begins.

I've spent a few weeks to get some simple forms working, where generally 10-20 lines of code done in a few minutes would be enough. I broke some pages. I got lost in documentation where vast amount of information lays around. I've hardly been able to understand that some paraghraphs that I've ignored as "wtf is this crap, where's the info?" were the data i've been looking for.

After enlightenment hit, i was still so far away from the solution.

Well, this rant is not about Twiki specifically, nor i am the one to comment it. I just want to criticise man's everlasting quest for "all-in-one that does it all and all and even all".

Well..uhm.. No!.. There's no such thing in real life, just get over and forget it already..

Allmighty is a fictional character from the worldwide bestseller series called Religion where "might" only comes from "magic".

Unlike the fictional character, when us humans try to have an allmighty solution, we just end up with useless crap or magical things that works somehow and only a chosen few can handle.

Twiki is a good example for this case. I am not one of that lucky few.

Learning the underlying magic took my weeks, and the best part of it is I have nowhere to use and profit from the experience i got. Lost time is lost.

Sigh..

Why can't we just have small lessmighty things, and make them collaborate when more is needed?

Tuesday, December 23, 2008

Can't start new thread

While trying to get some multitasking in python via using vast amount of threads, i've hit my head to "can't start new thread" problem.

At first, i thought that max open file limit was the problem, to verify i did a manual checking as seen below.


In [1]: count = 0
In [2]: from threading import Thread
In [3]: def blah():
...: pass
...:
In [4]: while 1:
...: Thread(target=blah).start()
...: count += 1
...:
...:
---------------------------------------------------------------------------
error Traceback (most recent call last)
/root/ in ()
/usr/local/lib/python2.5/threading.pyc in start(self)
438 _limbo[self] = self
439 _active_limbo_lock.release()
--> 440 _start_new_thread(self.__bootstrap, ())
441 self.__started = True
442 _sleep(0.000001) # 1 usec, to let the thread run (Solaris hack)
error: can't start new thread
In [5]: print count
438

438.. ôO'

i was expecting 1024 or so(mostly more than 1024 if running as root) but 438 was way lower than that. It seemed that another problem was going on.

 root@s1 [~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 65536
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

stack_size(10mb) seems to be the culprit here. stack_size is basically the maximum memory size a single process can allocate as stack.
10Mbs is much more than needed for my threads, so retrying the manual checking succeeded as expected.
 In [1]: import threading as t
In [2]: def blah():
...: pass
...:
In [3]: t.stack_size(128*1024) # 128K would be enough in the real code, so trying it here
Out[3]: 0
In [4]: count = 0
In [5]: while 1:
...: t.Thread(target=blah).start()
...: count +=1
...:
...:
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
.........

KeyboardInterrupt: # bored waiting :o/

In [6]: count
Out[6]: 68348 <------ Yays!

Now i'm happily enjoying my gazillions of semi-concurrent threads.

Tuesday, December 2, 2008

We hope so...

Hope is one of the main motives that keep us living.

If we have short term plans, we hope that they'll come true in as much as the same way we've thought. For long term plans, we hope that
not much of them will change, and if they change we hope them to be changed for good. If we don't have plans, we just hope something good will happen sooner or later.

We mostly keep living for the probability of winning, at the very least the probability of tomorrow will be better than today.

We also share our -mostly good- hopes with each other. Knowing that others have hopes for us, or at least just knowing that "others have hopes" motivates us, develops trust between people and strengthens relations.

We depend on our hopes in almost every fragment of our lives.

But let me strongly suggest keeping hope away from business. At least keep it away from being the only base for doing business.

Yes hoping is nice, keeps us fresh and tight and so...

Unfortunately, i feel old and trashed when getting excuses like

"We hope that electric shortout will end soon"
"Our customers delayed payments. We hope to get paid in -what ever-, and we'll pay you asap"
"We hope that this part will be completed in -whatsoever- by the third party and we'll be able to complete the work in time"
...

Hoping does not solve the problems about wrong decisions, promises about vaporwares or bad business models based on others' lousy plans.
One can only hope that everything goes as planned and expected when all that can be done is done.

I hope that i'll be able to keep hopes away from my business.

Monday, December 1, 2008

Multiple applications on localhost with lighttpd

I've been playing with Wiki applications for sometime. Well, to be honest, instead of "playing", "struggling" is much more suitable.

I've searched for hosting multiple applications by separating paths like $HOST/moinmoin/, $HOST/twiki/ etc.. But couldn't find useful tips around (or i'm not that good at Searching The Fine Web). Being tired of switching lighttpd configurations for each wiki setup, i've dived into configuration file and managed to make it run multiple wikis(one cgi, one fcgi) separated by pathnames.

Anyway, below is the snippet from my local server's lighttpd.conf. Somewhat messy, but works good for a few minutes work. http://localhost/twiki/ and http://localhost/moin/ redirects requests to said wiki engines.


$HTTP["host"] == "localhost" {
url.rewrite-once = (
"^/robots.txt" => "/robots.txt",
"^/favicon.ico" => "/favicon.ico",
"^/moin/moin_static173/(.*)" => "/moin/moin_static173/$1",
"^/moin/(.*)" => "/moin/wiki-engine/$1",
#"^/bin/?(.*)$" => "/twiki/bin/$1",
#"^/pub/?(.*)$" => "/twiki/pub/$1",
"^/twiki/(bin|pub)/(.*)$" => "/twiki/$1/$2",
"^/twiki/(.*)$" => "/twiki/bin/view/$1"
)
alias.url = (
"/moin_static173/" => "/usr/share/moin/htdocs/",
"/twiki/bin" => "/home/den/public_html/bin",
"/twiki/pub" => "/home/den/public_html/pub",

)
$HTTP["url"] =~ "^/moin/wiki-engine/" {
server.document-root = "/home/den/calisma/python/moin/"
fastcgi.server = ( "/moin/wiki-engine" =>
(( "docroot" => "/",
"min-procs" => 1,
"max-procs" => 2,
"max-load-per-proc" => 2,
"bin-path" =>
"/home/den/calisma/python/moin/bin/moin.fcg",
"host" => "127.0.0.1",
"port" => 3060,
"check-local" => "disable",
"broken-scriptfilename" => "enable",
))
)
}



$HTTP["url"] =~ "^/twiki/bin" {
cgi.assign = (
"" => "" ,
".pl" => "/usr/local/bin/perl",
".cgi" => "/usr/local/bin/perl",
".py" => "/usr/bin/python"
)

server.document-root = "/home/den/public_html/"
server.indexfiles = ("index.html", "index.pl",
"index.cgi")
}
}

About Me

hinoglu
Istanbul, Turkey
I'm a self-taught programmer in a developing country, trying to keep my business up and running.

I'll use this space as a sketchboard to improve my writing skills and a notebook for my adventures in business.
View my complete profile