What to do with the one-shot scripts

I can’t really answer this question, but I am really curious.

During work, I often have small(ish) tasks to solve, and when it is possible I rather write small one-shot scripts instead of doing some manual labor.

These scripts are rough, they are not made perfect because they don’t have to, they have one purpose, and they have to do it right. Sometimes they have small mistakes in them, sometimes they just aren’t perfect, sometimes they are awful. There are always compromises when creating these like ‘does this task worth that plus hours to put them, or just aim for the solvable problem’.

Let me show this in an example. Today I am working on a script what copies big amounts of data between two servers, I want to do it multithreaded, but I also want to limit the number of threads.

My quick and dirty solution looks like this:
for row in cursor:
while len(threading.enumerate()) > MAXTHREADS + 1:
time.sleep(.1)
worker = Worker(row)

Yes, I know, civilized people don’t use ‘sleep’, because that is a waste of CPU, but hey, I will run this script on an ephemeral EC2 instance, so I really don’t care. This will do it, so deal with it.

But after the migration, I do want to keep this script somewhere because it is possible that I’ll need something like this in the future, and I don’t want to reinvent the wheel then.

My GitHub account would be a perfect place for them, but I am not sure if I can put them there. I mean GitHub account is like a business card (remember Patrick Bateman), I don’t want to put ugly code them – I mean if I want to find a new workplace, then it will be possible that they will check my GitHub stuff, and I don’t want them to think if I am a bad coder.

If I had a paid account (which I don’t have) then I’d put these into a private repo.

Nowadays I put these into gists, but gists are not easy to search, so that is not perfect as well.

I am not sure what would be the best.

 

The way I like to compile my Go programs – Makefile

I was on the quest of searching the Holy Grail of Go programming, and I found something, which I doubt that it is, but close enough – for the first sight.

I have several problems with GO, first, that I write my code on an OSX box, and I’ll run the programs on Linux hosts, so I have to solve the cross compilation; my second problem with Go, that I don’t really like the “There is a GO project folder, and all the GO projects are relying on” approach. It makes using GitHub painful.

Continue reading “The way I like to compile my Go programs – Makefile”

My first impressions about Go language

I am fascinated. Maybe that should be enough, but I guess I have to write a bit more here because we are not on twitter.

I spent a few days to get know Go language, and now I am more than satisfied. I mean, all the project ideas which are floating in my head should be written in Go.

First of all, I have rewritten Mambo-collector to go (https://github.com/banyek/mambo) because I have faced some serious errors when I used it on CentOS 7 – I blame systemd -: If the process was running as root then after a few days of data collecting, killing that process was lead to restart the entire system, which is not a bug, it is a catastrophe. I tried to debug it several ways, but I am not sure where the problem is, it could be at the ‘loghandler’ redirection or any other place in python-daemon, or it is simply there is a buffer inside which overflows – I don’t really care, because mambo was just a proof of concept – what I used in production. I decided to abandon the project, and rewrite it in a language, what is designed for what mambo is: a system process with a high amount of parallel data processing. In the original version, I spawned an os thread for every query which will run (and we know that there is no real multithreading in python because of the global interpreter lock, at the end, building a multithreaded application in python is just an illusion. It is good for helping you to imagine what will happen inside your the computer, but you cannot expect real multithreading. It wasn’t a real problem because I only ran/run a few queries with mambo – replication lag, all processes inside mysql, active processes inside mysql – but if there would be a need to increase the number of queries that could lead to trouble. And it’s needless to say if you have a lot of time.sleep in your program, that is not the wisest use of your CPU resources: if a CPU is sleeping, then you waste resources.I have enough resources to let waste a bit, but it always disturbed me.

At the end, I decided to rewrite mambo in go.

Continue reading “My first impressions about Go language”