People. Tools. Processes. In that order.

That's my common response to the question "What are the most critical factors for software development process?".

People

Does anyone question that having good, skilled, motivated people is a good thing? I don't think so. But it seems that there is no such general agreement on the proposition "good people is the most important factor in software development".

Well, I will justify my assertion when describing the other two critical factors, noting why they aren't as important as having good people.

Now, the topic of how do you identify good people for software development, or what are the qualities that a good programmer must have, is beyond the scope of this blog post and already discussed endlessly on many places. Just find the criteria that works for you.

Tools

Good tools improve productivity, and avoid wasting precious human time with boring, repetitive tasks. Bad tools (or no tools at all) force people to have a hard time solving problems not very well suited for humans.

But they aren't more important than people, because good tools in wrong hands causes "interesting" results sometimes and good people without good tools will produce good tools anyway, given enough time.

In others words: good tools are made by good people.

Processes

Yeah, processes are useful. They formalize some common and/or wanted behaviour. Documenting them make the life easier for new people coming into your development team, and keeps things clear.

But they aren't more important than people. Throwing a process to a bunch of unskilled people will only help (if anything) to get predictable poor results. That may be better than plain old chaos, but it's not too high on my scale.

On the contrary, good people can see where you need a process and where you don't need it. Again, good people create good processes. I'd say that the better processes are the ones where you want to formalize and repeat an already successful behaviour.

Finally, it's better to have good people plus good tools than good people plus good processes. Believe me. Still not convinced?. OK, I was just trying to keep this section short.

It's better because processes are determined by the available tools. Sometimes a good tool makes a complete process irrelevant, or greatly simplify a process. And remember that a process tends to have a considerable amount of inertia so are difficult to change.

So, setting a process when there is no good tools in sight may make stupidly hard to adopt a good tool when it is available. Besides that, there is a good chance that what the process is doing is no more than a work around, not a proper solution.

Real case: some of the customers of the company where I work have really big codebases. To make such codebases somewhat manageable, they have setup scary processes, multiple environments that must be merged manually many times, and force providers to write documents explaining what artifacts are new, what is being changed and what should be merged and how. And they use (or misuse) ancient revision control tools. I admit it, their problem is hard, and just recently we are seeing tools that can manage a high amount of parallel changes to a big codebase (such as GIT, Mercurial, BitKeeper, etc).

But they focused on the wrong side of the problem, and tried to mitigate the observed symptoms instead of questioning the root causes and improving that. And now that the open source community have a myriad of tools that fixes the problem (or a big part of it) they will not be adopted on the short term because everybody is used to the stupid process and the fear of change is too high.

Some final words about processes: They should be "agile". Yeah, I know that "agile" is a buzzword, but I'm referring to agile software construction, opposed to the Big Design Up Front methodology. Processes should be iterated over and over again, because only in reality some problems (o potential improvements) will be obvious. And the smaller the process, the better. I mean, it's OK to have one process to do code reviewing, another to do major releases, other to deal with security problems and so on. But avoid throwing too many problems to one big process. It won't work.

I also planned to present a few examples where the people->tools->process work. But this entry is already too long so I will save the examples for a later entry.

Django on Jython: Status and Goals

I've been silent lately, mainly because I've been very busy with my job (My role in the company changed, and you know, such changes tends to carry a bit of chaos).

Anyway, I'm convinced that the unresolved things in the Django+Jython integration aren't that hard, so is only matter of time (and/or manpower) to get them fixed.

So let's review my original short-term goals:

1. The development server must run: Check (no reloading support yet).


2. The admin application must run: Check. This means that that we got working templates, views, models and backend, for this limited but very important test case.


3. The test suite must run: Well, it ran three weeks ago, but the lack of time.strptime on Jython caused the failure of one third of tests. Later, when applying a patch with a partial implementation of strptime, I got an obscure error from postgresql that killed the test suite (ERROR: failed to fetch new tuple for AFTER trigger [SQLCode: 0], [SQLState: XX000]). And now I'm getting unicode errors Update: Unicode errors fixed. They were caused by another unicode bug in Jython. So I'm again stuck with the PostgreSQL error.

While fixing the test suite crashes, I want to state some new goals:

1. Django must run inside a servlet container: Should be easy, thanks to modjy.


2. No failures on the test suite: This one is hard. Therefore, I expect to break it into sub-objectives as needed

Finally, if you want to play with Django on Jython without the pain of manually collecting and applying the necessary patches to both codebases, just download the modified sources using mercurial:

 $ hg clone https://hg.leosoto.com/django.jythonport
 $ hg clone https://hg.leosoto.com/jython.hacks

Software is Hard

Not exactly news, but a very good reading anyway: Software is Hard

Some remarkable quotes below.

About Software Estimation

After a half-century of software scheduling, after counting function points and lines of code and switching from waterfall development to spiral to agile methods, the most effective scientific tools we have for estimating software development time are:

• The Ninety-Ninety Rule : "The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time."


• Hofstadter's Law: "It always takes longer than you expect, even when you take into account Hofstadter's law."

About Software Construction

Every software engineer has a low opinion of the way we develop software. Even the term "software engineering," Rosenberg writes, is a statement of hope, not fact. He quotes the 1968 NATO Software Engineering Conference that coined the term: "We undoubtedly produce software by backward techniques." "We build systems like the Wright brothers built airplanes--build the whole thing, push it off the cliff, let it crash, and start over again." Certainly statements that could still be made forty years later.

About Software Complexity

...the nature of software is that the problems are always different. You never have to solve the exact problem that someone's solved before, because if software already existed that solved your need, you wouldn't have to write it. Writing software is expensive. Copying software is cheap.

Scott Rosenberg coins this as Rosenberg's Law: Software is easy to make, except when you want it to do something new. The corollary is, The only software that's worth making is software that does something new.

[...]

Fred Brooks said it twenty years ago in "No Silver Bullet" better than I can today: "The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence."

Javascript, Closures and Parameter-passing Styles

Yesterday, a friend asked for help with this Javascript code, which didn't work:

    for (var i = 0;i < menu.childNodes.length; i++) {
        if (menu.childNodes[i].nodeName == "SPAN"){
            var element = menu.childNodes[i];
            connect(element, "onclick", 
                    function(){showItem(element)});
        }
    }

The observed behaviour was that clicking on every sub-menu item always showed the latest sub-menu, instead of their own sub-menu (connect is a MochiKit utility function, that among other things, acts as a cross-browser addEventListener). The bug is subtle. I will explain it after showing one possible solution:

    var makeShowItemEventListener = function(element) {
        return function(){ showItem(element) };
    }
 
    for (var i = 0;i < menu.childNodes.length; i++) {
        if (menu.childNodes[i].nodeName == "SPAN"){
            var element = menu.childNodes[i];
            connect(element,"onclick", 
                    makeShowItemEventListener(element);
        }
    }

This works. Why?.

The function makeShowItemEventListener receives a element and returns a function that, when invoked, will show that element (it's a so-called high-order function). So, what's the difference with the previous code, which coded the function directly?

It's all about how the element variable is "passed". On the first code, the anonymous function is a closure, which captures the variable element from its scope. But it captures a reference to the variable, so when the variable changes, the referenced variable inside the closure also sees the change. In other words: they are the same. So all event listeners are bound to the very same element variable which obviously ends pointing to the last sub-menu when the for loop finishes.

The fixed code passes the element variable to another function, and on Javascript, the parameters are passed by value, as on Java, C#, Python, etc.. So, inside the makeShowItemFunction, the formal parameter element is a copy of the original reference, and doesn't change if the actual parameter element is modified.

Finally, considering that MochiKit was already included, we didn't need to make a specialized function which only purpose is to return another function with a fixed parameter. That's what MochiKit's partial() does. So the resulting code is:

    for (var i = 0;i < menu.childNodes.length; i++) {
        if (menu.childNodes[i].nodeName == "SPAN"){
            var element = menu.childNodes[i];
            connect(element, "onclick", 
                    partial(showItem, element));
        }
    }

This is an example of why it's important to know and understand your tools, not just using it blindly. It will pay on the long run, not only when debugging problems like the one shown here, but also when developing way better solutions, because you know a better tool for each job.

Don't talk about "layer n"

I don't know if it's common to refer to multi-tier architecture layers by their numbers. I mean something like: "Argh, the date is presented in a reverse format, it must be a layer-one bug", or "The layer two seems to be really inefficient". Well, I know many people who uses that jargon.

If you are among them, please make me a favor: don't use numbers to identify software components or layers. Not when talking to me at least.

Why?

• You don't really save much saying "layer one" instead of "presentation layer", "layer two" instead of "application layer" and so on.


• I don't know how many tiers or layers you are supposed to have, so I'm unable to make sense of any number greater than one.


• I'm stupid, so I always forgot if the first layer is the user interface or the more basic layer which support everything else. So even "layer one" is meaningless for me.


• I'm not impressed by the words "layer" or "tier". Not even if they are followed by a number.