webfactory Neuigkeiten

Yes, we geek or A Tale of Mincemeat and Zucchini

webfactory GmbH — Thu, 09 Jul 2009 00:00:00 +0200

Yesterday we (dr, hs and sb) were working late, and around 10:00pm lunch seemed very long ago indeed. A quick survey of the kitchen confirmed that we had enough leftover ingredients for spaghetti bolognese with onions, tomatoes and zucchini. What we didn't have was mincemeat and somebody willing to leave his desk long enough to cook.

A short IM battle ensued.

sb: go and cook us dinner :)
dr: you two designers.. me one programmer..
dr: is this democracy?
sb: yes
dr: this is oppression
sb: yes
sb: NOW COOK
dr: NO!
sb: NO YOU COOK
dr: we fight for our freedom!
sb: COOK!
dr: permission denied
sb: COOK!
dr: permission denied
sb: COOK!
dr: permission denied
dr: you don't get it..
sb: COOK!
dr: permission denied
dr: hmm.. maybe I should say: "insufficient rights"
sb: sudo cook dinner -hs -sb
dr: damn. I'll do it...
dr: buy mincemeat!
sb: what! we'll eat vegan
sb: we have zucchini!
dr: sudo buy mincemeat
sb: shit.
dr: pwnage
sb: Enter password:
dr: mincemeat
sb: Sorry, try again. Remaining tries: 2
dr: ahh.. mom:
dr: ********
sb: SHIT

The quite enjoyable dinner was served soon after.

About the TCP MSS and wrong checksums

webfactory GmbH — Fri, 06 Mar 2009 00:00:00 +0100

During the last days, I had to refresh my knowledge about the gory details of the TCP protocol. By far I'm not versed enough to get into something like this or that, but at least my current problem was solved.

I spent a few hours troubleshooting a strange TCP connection problem. HTTP connections to one particular host were no longer working since we got a new Cisco ASA 5510 firewall managed by an external provider, set up to NAT outgoing connections. Initial connections to that host and the first HTTP request returning a redirect worked, but the following request would only return a few KB of HTML and then stall. Only that host was affected, at least we had not observed the problem somewhere else.

I did a tcpdump of the traffic between the firewall and our SDSL uplink and got the following.

mp@blackbook:~$ sudo tcpdump -v -ttt -s0 -n host remote.host
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
000000 IP ([...], length 48) firewall.26167 > remote.host.80: S, cksum 0x0b7b (correct), 3325973421:3325973421(0) win 65535 <mss 1380,nop,nop,nop,nop>
213304 IP ([...], length 44) remote.host.80 > firewall.26167: S, cksum 0xe020 (correct), 83736657:83736657(0) ack 3325973422 win 32768 <mss 1380>
000320 IP ([...], length 40) firewall.26167 > remote.host.80: ., cksum 0x778e (correct), ack 1 win 65535
000555 IP ([...], length 396) firewall.26167 > remote.host.80: P, cksum 0x3f23 (correct), 1:357(356) ack 1 win 65535
225193 IP ([...], length 317) remote.host.80 > firewall.26167: P, cksum 0xdd51 (correct), 1:278(277) ack 357 win 32768
017471 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
016391 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x766d (incorrect (-> 0x766c), 1658:3038(1380) ack 357 win 32768
203816 IP ([...], length 40) firewall.26167 > remote.host.80: ., cksum 0x762a (correct), ack 278 win 65258
226371 IP ([...], length 1420) remote.host.80 > firewall.26167: ., cksum 0x6120 (incorrect (-> 0x611f), 3038:4418(1380) ack 357 win 32768
2. 398071 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
2. 947090 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
5. 479496 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768

That looked as if the remote host was sending a lot of packets with wrong TCP checksums. Our firewall would just trash those packets, not sending out ACKs for them and wait for a retransmission. Those retransmissions can be seen at the end of the trace and trickled away with geometrically increasing intervals. As the retransmissions show the same error, the bottom line is that nothing gets along anymore.

Interestingly, the checksums were always off by just one. Had the packets really been altered (for example, corrupted by bit errors on the wire) during transport, the deviations should be arbitrary.

So my first guess was that the remote host or some component along the path had a bug in the procotol stack and was putting in incorrectly calculated checksums. But OTOH it's hard to imagine that we would be the first to discover such a bug.

When taking the firewall out of the mix the connections worked fine and all packets were ok. This is strange, as the TCP checksum is calculated based on a single TCP packet with some additional information from the IP layer. So as to the correctness and verification of the checksum, it is irrelevant what has been sent in the other direction before and whether such previous packets have passed the firewall or not.

Now one important observation was that the incorrect checksum was only in packets that were 1420 bytes in total size. This is the maximum possible size one can observe in this connection for a MSS of 1380 is negotiated during the TCP handshake and 40 byte for IP and TCP header come on top of that.

Using a test host placed outside (before) the firewall would result in working connections with an MSS of 1460, which is the Ethernet MTU of 1500 minus the 40 bytes. Setting the MTU to 1420 on that host yielded a MSS of 1380, but the returning packets were correctly checksummed in this case.

So although the MSS seemed not to be a sufficient condition to trigger the problem, I started to investigate why the firewall would use a value of 1380 where 1460 should be possible. We found a setting called "force maximum segment size for TCP proxy connections" in the Cisco's admin interface that was set to 1380. We unset that option and voilà, everything worked with the MSS going up to 1460.

What rankles me is that although the symptoms are cured, I still haven't found the cause for the wrong checksums. I don't see why setting the MSS to a lower value than necessary should be a problem at all (performance issues aside). And, having the checksums always off-by-one smells like another bug somewhere along the connection's path. I googled a lot for this one, but the only relevant results were in a discussion in the context of SMTP (but that's TCP after all) over there and there.

Hopefully, this article will save someone some headaches. If you have any ideas regarding the cause of this problem, I'd be glad to hear from you. Feel free to leave a comment!

wfDevCamp 2008: Work Away From Work

webfactory GmbH — Wed, 28 Jan 2009 00:00:00 +0100

Sometimes, special tasks require unusual measures. It was obvious that we needed to move the development of wfDynamic, webfactory’s content management framework, a big step forward. So in December 2008 the webfactory team headed to an apartment in the Austrian mountains for a week of focused, uninterrupted work.

In the middle of daily routines, it is difficult to bring all the people together that are needed to achieve giant leaps in product development. At a small software development company like webfactory, everyone is working on different projects with different rhythms. There are meetings with customers, impromptu meetings of small sub teams on project-related minutiae and phone calls that need to be handled.

But we needed to get things done: wfDynamic heavily relied on user interface techniques that didn't comply to today's web standards and accessibility guidelines. Replacing these techniques by state-of-the-art alternatives was a job that needed time, focus and every skill we had on the team.

We felt that a week away from the office with the whole team would be the break we needed. Enough distance from „business as usual“ to really be able to zone into product development. The idea of a webfactory DevCamp was born.

Laying the groundwork

The first step to take was finding a location. We agreed on a few particular requirements:

far enough from home to have a clean cut from everyday life
no distractions (no phone, no emails)
a flexible and inspiring working environment for the team
attractive options for breaks

We were very fortunate to find a place that almost perfectly met our requirements in the Austrian Alps. One team member's family had been using a holiday apartment in Westendorf, Tyrol for years. The apartment, situated on a small mountain road, had a good size to accomodate our company of five. The fresh mountain air and great views over the valley from our balcony provided an inspiring background for a concentrated working atmosphere. A ski lift in walking distance and many hiking trails through the snow-covered forest behind the house were just calling for attention whenever we needed a longer break. However, most of the time we were too captivated by our ideas and discussions to even consider going outside.

The setup

Westendorf is about 700km south of our office. We decided to hire a van for the trip to foster team communication even on the journey.

On arrival in Westendorf we hit the next „Hofer“ discount market to buy some food and a „yesss“ mobile internet flatrate with USB stick. Equipped with the USB stick, a Mac Mini was serving as our router for internet access and also as host for our version control repository. It took a while to set up, but after we got it running it didn’t let us down once.

Discussions

The team that set to work reinventing our product was quite diverse. Besides the more obvious division in software developers and interface designers, there were also subtly different flavours to each member. Analytical minds mixed with visual thinkers, the urge to save time for careful consideration clashed with pragmatic hands-on approaches. The question to what extent we would reuse the old code or start from scratch was another controversial subject.

All of it led to some very difficult but interesting and productive discussions throughout the week. Incidentally, it took more than half a day before the developers touched the first lines of code. The designers dug into Photoshop to knock up a few visual ideas while the most fundamental code discussions were under way.

Iterative Improvements

It was interesting to see how we tended to discard the previous day’s ideas every morning because someone had come up with an even better approach after a break and/or getting a few hours of sleep. At last, Day 4 of our stay saw the first implementations of our new designs.

Sum of the parts

After getting a better idea of the user's flow through and interactions with our application, the designers went back to their wireframes and started implementing them based on growing HTML outputs from the development faction. It worked quite smoothly and we had a rudimentary system up and running on the last day. With a lot of work left ahead, but a solid new code foundation under our belts, we returned home in high spirits.

Conclusion

The webfactory DevCamp exceeded all our expectations. The development results were fabulous and we are absolutely convinced that we could never have achieved anything remotely similar during regular hours at the office. The event was also a great team building experience - shared memories of our day out (braving the slopes on sleighs) and of a few quite effective discussions during the shorter hikes around the house top the charts next to the energetic and productive atmosphere during development.

We will definitely do this again!

Preach What You Practice

webfactory GmbH — Tue, 07 Oct 2008 00:00:00 +0200

webfactory has always been about creating highly usable and accessible websites, keeping a close watch on web standards and the POSH paradigm. This September we finally decided it was time to start preaching what we practice.

So you say, „Dude, get real! It’s 2008! Web standards are not only known among the web working crowd, they’re our bread and butter!“. Sadly, there are still websites prooving you wrong. There is still need to spread the gospel and point both established and new web workers in the right direction.

On a sunny Saturday afternoon we invited a group of students who are serving apprenticeships as digital media designers like webfactory’s own Søren Birkemeyer into our company's Bistro and set about to highlight the advantages of semantic HTML and web standards. After an initial phase of explaining concepts and best practices, we confronted the group with a real-world task: translating a finished design (a Photoshop file) into HTML and CSS.

With one exception, all attendees had a print-related background in their companies and were rather new to both HTML and CSS. And yet, after only half an hour, people got to grips with our editor of choice for the workshop and the first discussions arose about correct semantics and markup. „Is this a headline at all?“ „I have ‚by’ and ‚from’ looking special to me in this sentence, but if I emphasize them that’s not what really want, or is it?“

With regard to the exemplary weather we had been planning for a quite open workshop, hoping for perhaps two hours of concentrated work before the session dissolved around beers and a game or two on our kicker table. Alas, we ended up closing the door behind the last die-hards after seven hours of hard work, coffee and beautiful code.

We are really happy about the turnout and will certainly continue with another workshop in the near future, so stay tuned! After all, who could resist the chance to turn people into POSH-nerds? :-)

Browsertesting can be fun!

webfactory GmbH — Mon, 18 Aug 2008 00:00:00 +0200

Okay, enthusiasm may have gotten the better of me when I was punching in this entry's title. But even when you're doing what every frontend engineer hates by default – testing and retesting your page in the major browsers known to mankind – there are a few tools that can really make things easier for you.

Let us revisit a great talk by Nate Koechley first. I saw him at this year's @media conference where he went and instilled some (needed?) pride into those of us who work the (X)HTML/CSS/JS/DOM/Browser angle of websites. Is frontend engineering a "real" job? Hell yeah! Check out the slides of his talk on his blog and see how he arrives at the magic number of 672 different situational combinations of website/user environments that we have to prepare and test for.

So what can ease the job with the actual testing work? It goes without saying that you should always develop with at least three to four browsers, namely the current builds of IE7, FF3, Opera and Safari, open next to your IDE. It's miles better to check browser behaviour immediately than finishing off a template for Firefox, opening it in IE and having to start all over again.

I have the advantage of working on a Mac with Windows XP running on a virtual machine, so I can cover these two operating systems and the main browsers with relative ease. But what about testing multiple versions of the same browser?

Testing multiple Internet Explorer versions in Windows: IE Tester

Up until recently my favourite tool for testing multiple IEs was the aptly named Multiple IE. However, when I ran into problems with testing browser specific styles (via conditional comments) I went looking for an alternative and found IE Tester. This one is working flawlessly so far and even includes a IE8 beta engine for pretesting, if you're so inclined.

Testing multiple Firefox versions: MultiFirefox

Since Firefox is Firefox you'd guess it's fairly easy to set up two parallel installations, right? Well, it's not quite that simple, since FF has a tendency to overwrite itself and/or its user profiles when you install a second version.

For Mac users there's a pretty straightforward solution called MultiFirefox that handles profile management for you. Install FF3 and FF2 and the application will take your hand and help you set yourself up.

Windows users have to go through a bit more manual tuning work, which is nicely covered by Command-Tab.com. They also mention MultiFirefox as the tool for MacOS.

If you have better solutions for browser testing, or just found these recommendations to be helpful, don't be shy and drop us a comment!

PHP Namespaces Explained

webfactory GmbH — Sat, 17 Nov 2007 00:00:00 +0100

Namespaces are one of top new features you can expect from the upcoming major PHP versions. There's still a lot of discussion and confusion about how it's going to work in detail, especially because official documentation is scarce and behaviour is subject to change. That being said, in this article I will try to sum up what I could observe playing around with a fresh checkout from PHP's PHP_5_3 CVS branch.

Namespaces Really Are Packages

PHP 5.3 brings the new "namespace" statement. When present, this statement must be the first one in a file. Thus, you cannot use multiple namespace statements per file :-) or even nest the statements. This is why PHP namespaces are more like Java's packages than the namespace concept as in, for example, C++. Anyway, I will use the term "namespace" here. Now what does happen?

test1.php

namespace webfactory::NS1; class X {} print get_class(new X()) . "\n";

Basically, the namespace statement declares that all class names you define in that very file belong to the namespace. Namespaces also work for function names, but I won't cover functions any further here. The above example will print "webfactory::NS1::X" as that is what the class X is really called. This is said to be the fully qualified name of the class.

"Defining" a class name means actually providing a class definition. If you just require() a class definition from another file, that class won't end up in your file's namespace. It will be in the namespace set for the file where the class is defined (if any is set).

In test1.php, "webfactory::NS1" is a compound name. You may use "webfactory" as well as "webfactory::NS2" or even "webfactory::NS1::Sub1" as namespaces in other files. In all these files, you can define classes named X and they all will be discriminable because their fully qualified names differ. If you declare a class X in a file without a namespace statement, the fully qualified class name will be "::X" (however, get_class() will just return "X" in that case).

For the rest of this article, let's assume we've got the following two files using the two namespaces webfactory::NS1 and webfactory::NS2:

ns1_x.php

namespace webfactory::NS1; class X {}

ns2_x.php

namespace webfactory::NS2; class X {}

Attention Attention...

Before we proceed, understand that the entire namespace thing is just about naming and referring to classes in a convenient way. It has nothing to do with packaging or loading classes for you. You still will have to use require() and its friends to load your classes. If __autoload() does that job for you, you can probably benefit from tweaking your handler to make use of the namespace names, but basically that changes nothing.

Fully Qualified Names

The most straightforward way of telling PHP what you mean is to refer to classes by their fully qualified names. You can always do that regardless of the namespace you're in or the namespaces you use. Example:

test2.php

require("ns1_x.php"); require("ns2_x.php"); print get_class(new webfactory::NS1::X()) . "\n"; print get_class(new webfactory::NS2::X()) . "\n";

No surpriseses here. This will print "webfactory::NS1::X" and "webfactory::NS2::X", the fully qualified names of the classes.

Prior to having namespaces, this example would have caused an error on the second line as the class name X was already taken. With namespaces, both classes can coexist and by using fully qualified names you are able to tell both apart.

Now that was not all many people have been waiting for so long... You could always have prefixed your class names and have them called webfactory_NS1_X and webfactory_NS2_X respectively to avoid such kind of trouble. But that was often too clumsy to type and read, so many people simply ignored the problem. Namespaces to the rescue!

Using Namespaces

Namespace support also brings the new "use" statement. It takes the form of "use A::B::C::D" or "use A::B::C::D as Z". If you omit the "as" part, the last name part will be used instead, so "use A::B::C::D" is equivalent to "use A::B::C::D as D". The use statement may be placed anywhere in the global scope (that is, not inside classes or functions) and takes effect from that point on to the rest of the file – but only in that file!

Think of "use" as a statement to create a per-file shorthand or alias map. When writing "use A::B::C::D as Z", an entry will be made for your current file that maps Z to A::B::C::D. Think of an associative array like array("Z" => "A::B::C::D")!

From that point on, whenever you use Z somewhere in your file to qualify a name, Z will be looked up in the map and A::B::C::D will be used instead, eventually adding further qualifications that followed Z.

You can use that to refer to classes from other namespaces, to simply shorten qualifications or to resolve name clashes at the point where you bring conflicting names together. Here are some examples for each case:

test3.php

use webfactory::NS1::X; require("ns1_x.php"); print get_class(new X()) . "\n";

When creating a new X instance, this will first look up "X" in the alias map and find "webfactory::NS1::X" which is the fully qualified name for the class defined in ns1_x.php. Whether or not there is a namespace statement in test3.php is of no relevance here. Cases like this one always have a fully qualified class name on the use statement.

It would also not make a difference if you changed the order of the first two lines. Remember, name aliasing is independant of loading class definitions and it's not unless the third line (the "new" statement) that both must have been taken care of.

test4.php

use webfactory::NS1; use webfactory::NS2; require("ns1_x.php"); require("ns2_x.php"); print get_class(new NS1::X()) . "\n"; print get_class(new NS2::X()) . "\n";

This will put two mappings into the alias table. "NS1" maps to "webfactory::NS1", so "new NS1::X" is just shorthand for "new webfactory::NS1::X". The same applies for NS2.

If a use statement refers to a namespace name and creates an alias for it, you will always have to use an additional qualification like "alias::classname" to refer to a class. In the last example it was obvious that a qualification through "NS1::" or "NS2::" was necessary to make clear which class is meant.

Please note that "use webfactory::NS1" does not make all classes defined in the webfactory::NS1 namespace available. As said above, PHP has no idea of your codebase and the classes you might have defined in any given namespace. So you cannot import all classes from a given namespace with a single use statement.

test5.php

use webfactory::NS1::X; use webfactory::NS2::X as AnotherX; require("ns1_x.php"); require("ns2_x.php"); print get_class(new X()) . "\n"; print get_class(new AnotherX()) . "\n";

This is the pattern to locally resolve name clashes. Assume you need to work with two external libraries you have no control over and both define a class X you want to use. Before namespaces, you were lost. Now, both libraries hopefully will use different namespaces (webfactory::NS1 and webfactory::NS2 in this case). The use statements now declare that X is the class X from the webfactory::NS1 namespace and that AnotherX is X from webfactory::NS2. If you understood the aliasing mechanism, you're probably not much surprised.

The resolution is "local" in that it only applies to the test5.php file. Other files won't be affected and if you just need to use (for example) the webfactory::NS2::X class somewhere else, it is perfectly ok to just "use webfactory::NS2::X" and create a "new X" there (or simply use the fully qualified name, of course).

Clashing Aliases

When you try to put something into the alias table and the key is already given, you get a name conflict. The lookup key must be unique because otherwise when using it it would not be clear which definition is to be used. Some examples:

test6.php

use foo::NS1; use bar::NS1;

You will get an fatal errror "Cannot use bar::NS1 as NS1 because the name is already in use". In this case the ambiguity even is in finding the NS1 namespace: "NS1::X" might refer to foo::NS1::X or bar::NS1::X. Similar one:

test7.php

use foo::NS1::X; use bar::NS2::X;

In this case, X refers to a class but it is unclear to which one. Remember, using namespaces has nothing to do with loading class definitions. You could require() both files that define X in the foo::NS1 and bar::NS1 namespaces and it would be no problem at all because both definitions differ in their fully qualified names. It's the use statement itself that creates the conflict as both statements want to use the same key in the alias map. In test6.php and test7.php, I do not even require the class definitions and the alias map is never looked at for name resolutions.

Names in Namespaces

Now here comes an extra rule. Maybe it is not what really happens under the hood, but the explanation closely matches what can be observed.

When you put a "namespace A::B" statement on top of your file and define a name Z in it (remember my definition of "defining" from above), an alias "Z" for "A::B::Z" will be put into your file-local alias table. That allows you to use the new name right away – and that is how my example test1.php way up worked: By defining class X in namespace webfactory::NS1, we implicitly ended up with a X-to-webfactory::NS1::X mapping in that file.

Together with the preceding section, you should understand that the following will cause a fatal error on the third line because the "extra rule" tries to add an alias for a name that is taken.

test8.php

namespace webfactory::NS1; use webfactory::NS2::X; class X {}

The Edge Cases

Re-read the extra rule and take it to the limits: The alias is made up when you define a name and it is put into the alias table for the file the definition takes place in. So what happens when the definition is sourced out to another file? Consider:

test9.php

namespace webfactory::NS1; require("ns1_x.php"); print get_class(new X()) . "\n";

For the ns1_x.php file, the extra rule will create an alias. But as the alias table is per-file, in test9.php there is no X entry.

Now what happens when a new X is created? Blurring the difference between defining a class and requiring() the definition, the following extra-extra rule kicks in:

First, the engine checks if X is a known alias – here it isn't. It then considers X as part of the current namespace. That is, it checks if a class with the fully qualified name "webfactory::NS1::X" has been defined somewhere. This time, it will find and use the definition from ns1_x.php.

A Kind Of Magic*

I hope that the following will be a pathological case in practice, but the result may be surprising enough to be documented.

test10.php

namespace webfactory::NS1; use webfactory::NS2::X; require("ns1_x.php"); require("ns2_x.php"); print get_class(new X()) . "\n"; print get_class(new webfactory::NS1::X()) . "\n";

This is basically the same as test9.php, with one subtle difference. This time we "use webfactory::NS2::X" to make up an alias table entry for X. In this case, the extra-extra rule does not trigger and you will get "webfactory::NS2::X" for the first print. Creating aliases with "use" takes precedence over using names from the same namespace when these are defined in another file.

If you had written "class X {}" instead of require("ns1_x.php") – and ns1_x.php does no more than that in the same namespace as test10.php – you'd get a fatal error because then the extra-rule would try to create the alias for X and finds that name taken by webfactory::NS2::X.

Final Remarks

When you got used to using __autoload() or the SPL equivalents to load in your class definitions, regardless of the namespaces defined or used in your files you will be passed the fully qualified names of the classes you need to load. For all the examples I provided here that is always what get_class() returned. If you omitted the require() calls from test10.php and had set up an __autoload handler instead, it would have been called for webfactory::NS2::X and webfactory::NS1::X on the first and second print line, respectively.

When using variable class names as in "$foo = new $bar()", where $bar holds the name of the class you want, name resolution rules don't apply so you will have to work with fully qualified names. Stanislav Malyshev is unsure if that is a shortcoming. (A short sidenote on this pattern: Interestingly, your __autoload() handler will be passed the value of $bar, no matter if it's a syntactically valid class name. That might be an interesting remote code inclusion vector if your handler implementation is too naïve >:-)).

Although I always used the "new" operator for demonstration purposes in this article, the same rules for resolving names apply in other cases where you refer to class names – think of type hints in method signatures, for example. The serialize() function will use the fully qualified class name as well to allow for correct de-serialization later on.

I hope this article helped you to understand how or why things work as they work. If you found it helpful or have any remarks, I'd be glad if you leave a comment.

Update 2007-12-06

Currently, there's a lot of discussion on the PHP Internals Mailing List about wheter or not the current namespace implementation makes sense and solves the problems people have in practice or if it would be wise not to ship it at all. So before you start updating your codebase, relax and wait how things evolve - you still have plenty of time before PHP 5.3 will be available and used at a large scale. At last, the first official documentation is available.

(* Courtesy of Queen, 1986)

Dokumentenmanagement in einem Tag

webfactory GmbH — Thu, 08 Nov 2007 00:00:00 +0100

Nach 10 Jahren hatten wir genug von der Papierflut. Jede Eingangsrechnung kopieren und in verschiedene Ordner sortieren, das gleiche mit Ausgangsrechnungen. Umlaufmappen, Kürzel, Chaos auf dem Schreibtisch. Das muss doch besser gehen, dachten wir uns.

Zunächst war klar: Ein Scanner muss her. Denn Post kommt ja nach wie vor auf Papier. Nach kurzer Recherche in der c’t stießen wir auf den Canon ScanFront 220. Ein Standalone-Gerät mit integriertem Touch-Screen, das auf Knopfdruck Scans gemäß vordefinierten Jobs über Netzwerk als E-Mail verschicken oder auf FTP- oder SMB-Ordnern ablegen kann – als durchsuchbare PDF-Datei.

Und plötzlich kam die Erkenntnis: Wir haben eigentlich alles, was wir brauchen, um loszulegen. Ein bisschen Risiko war dabei, denn der Scanner kostet an die 2000 €. Doch die Aussicht war einfach zu verlockend.

Der Plan: Der Scanner steht da, wo die Post ankommt. Jedes Dokument wird sofort gescannt und in einen FTP-Ordner auf einem Linux-Server abgelegt. Dort findet ein Cronjob die PDF-Datei, verschiebt sie ins Archiv und legt einen Eintrag in einer Datenbank an. In diesem Eintrag werden alle Meta- und Verarbeitungsinformationen zum Dokument abgelegt. Das Papierdokument kann sofort in einem Schuhkarton für den Aktenkeller abgelegt werden, denn es wird nicht mehr benötigt.

Für die weitere Verarbeitung des elektronischen Vorgangs in der Datenbank setzen wir wfDynamic ein. Es kann von Haus aus mit unterschiedlichen Typen von Datensätzen umgehen (Multi Table Inheritance). Werden zu einem Basisdokument z. B. im Wesentlichen das Datum des Dokuments, der Scanzeitpunkt, eine Bezeichnung und die Zuordnung zu einem Absender/Empfänger (Kunden, Lieferanten, Behörde, …) gespeichert, so enthält eine Eingangsrechnung zusätzlich die Rechnungsnummer, den Rechnungsbetrag und Angaben zur Buchungsart (kreditorisch, Kasse, Kreditkarte).

Spätestens an dieser Stelle fing es an, Spaß zu machen. Wir haben bereits den Absender der Rechnung, das Datum, einen Beschreibungstext, den Betrag erfasst – was läge näher, als diese Angaben direkt in der Buchhaltung weiterzuverwenden? Die Adressen, zu denen wir die Dokumente zuordnen, stammen direkt aus unserer Buchhaltungssoftware. Wir kennen also zur Adresse auch das Kreditorenkonto. Nichts einfacher, als alle Daten in eine CSV-Datei zu packen und in die Buchhaltung (in unserem Falle Lexware Professional) zu importieren…

Es ist 21:30 Uhr. Um 11 Uhr morgens ist der Scanner gekommen. Der erste Stapel Eingangsrechnungen hat seinen Weg vom Scanner über die wfDynamic-Datenbank in die Buchhaltung gemacht und ist dort erfolgreich verbucht worden. Morgen müssen erstmal wieder die Kunden bedient werden – aber die Lösung ist bereits funktionsfähig und im produktiven Einsatz.

Jetzt werden wir Erfahrungen sammeln – und dann an dieser Stelle wieder darüber berichten.

What 64 bits can do for you

webfactory GmbH — Fri, 28 Sep 2007 00:00:00 +0200

Recently we bought a DELL PowerEdge 2950 and meanwhile deployed it for our web hosting service. We installed Debian Etch, the first Debian release that officially and fully supports AMD64. So we had to decide were able to choose between installing a 32 or 64-bit version of our favourite OS. Now what is that decision about?

Intel vs. AMD

As you might know, already a few years ago Intel tried (with the help of Hewlett Packard) to develop a new "Intel 64-bit Architecture" (IA64), specifically for their customers asking for more computation power and more memory. However, the problem with IA64 is that it is incompatible with the "traditional" architecture and instruction set known as x86, requiring vast changes to make operating systems, compilers and applications work with it. Maybe not surprisingly, IA64 never gained the importance Intel hoped.

AMD, on the other hand, carefully chose to extend the established x86 architecture to make it 64-bit capable. They made registers 64-bit wide, added a few more of them to the processor and came up with some new CPU instructions to leverage 64-bit computation operations. Contrary to IA64, this architecture named AMD64 still supports the x86 instruction set which basically means that it is capable of falling back into 32-bit mode and run "old" code without modifications. AMD64 was rewarded with a lot of acceptance in the market Intel aimed at and eventually Intel chose to license some of AMD's ideas. Now more recent Xeon processors are "EM64T" capable where EM64T is the "extended memory 64-bit technology" which basically is Intel's name for the same thing.

If you're unsure whether your machine supports EM64T, cat /proc/cpuinfo and look for the "lm" flag. Now what difference does that all make?

64 > 2 * 32 ?

Switching to a 64-bit OS doesn't magically make your applications faster. Most probably you will benefit if you're doing complex numerical calculations or simulations and your software is capable of (has been optimized for) using the 64-bit instructions, allowing it to calculate or transport "more" data per CPU cycle.

Often the more interesting aspect of running a 64-bit OS is address space. With 32-bit registers, you can only store 2³² distinct values. If these values are memory addresses, you will end up with a 4 GB address space a single process can use. Once upon a time, "640 k ought to be enough for anyone", but nowadays, especially with large databases you want to be able to access more than 4GB of memory — mainly to cache data in memory instead of reading it from disk when needed. So 64-bit is a good thing, right?

But...

Now here's the "but": There are some details that should not hinder you from switching over to 64-bit, but you can save yourself (or your overworked sysadmin) some time and headaches if you consider them first. As you will see, these are merely practical aspects that will vanish as 64-bit systems are getting more and more commonplace.

An application must (of course) be compiled for 64-bit to make use of it. The same holds for all the libraries your application depends upon - when the app is 64-bit, the lib must be it as well. Now that may be a problem if either of them is not available for the AMD64 platform, maybe because it's closed-source or a vendor-provided device driver that has not yet been 64-bit-proofed.
Debian provides x86 and amd64 versions of a package and puts 64-bit libraries into /lib64 which /lib is a symlink to.
Now when you've got a 32-bit application binary, you can run it on an amd64 arch system (Surprised? Scroll up!) as long as you got the 32-bit versions of all the libraries it depends on installed as well. On Debian, you can install the ia32-libs package that contains 32-bit versions of some often-used libraries. The dynamic loader will figure out that you're trying to run a 32-bit application and load the libraries from /lib32 which is a symlink to /emul/ia32-linux.
Trouble starts when you need to install the 32-bit version of a certain Debian package on an amd64 arch system or even need to add 32-bit versions of some libraries yourself. At that point you're pretty much lost because all that will no longer work under the control of the APT package management system, so you will have to care for dependencies and upgrade tracking yourself.
The other way round, 64-bit binaries do not work on 32-bit (x86) systems. So what?! Why am I telling this?
Maybe you have been using Debian long enough so you learned that the versions of software packages you "officially" get are somewhat... err... older. Debian "stable" means stability in version numbers, not stability through most recent versions. And basically, this is a good thing because you don't really want to change library versions too often on a production system as every such change brings the risk of regressions in behaviour.
However, in the past we had a few situations where we absolutely needed to get a certain bug fixed in the libc, libxml2, MySQL or just had to keep up with the current PHP versions. So we started to maintain some variants of the Debian packages ourselves and set up a server to deploy these variants to all our machines leveraging the APT package system. (Maybe interesting enough for another article?) Now running amd64 on some machines means having to build all these packages twice, once on x86 and once on amd64 (did anyone say "cross-compile"?) and manage both architectures in our package repository — quite some extra work.
It may be that application developers made assumptions about the size of some structures (pointers or ints) that held true on the x86 platform but are wrong in the general case. That may lead to bugs that suddenly surface in applications and libraries you have been using for a long time without problems on x86. Sara Golemon told such stories at the PHP conference 2006 code camp: It was a single wrong data type declaration in the PHP innards that caused days of tracing and debugging when they moved some apps to 64-bit at Yahoo!
It may take some time for applications to become as mature as they are on x86, so keep this in mind.
64-bit applications tend to be somewhat larger and have a bigger memory footprint, especially because pointers are larger (finally, this is what it's all about :-)). While size on disk should not really be an issue, the in-memory size may adversely affect the efficiency of your precious level-2 cache.

Conclusion

Clearly, the future will belong to 64-bit systems. But it's up to you to decide whether you want to start the transition right now — operating systems support it — or still wait a little. Before you step up, be aware of the differences and the problems you might get.

Especially, save yourself some grief and don't change just for sake of change. Find out what your applications need — are you number-crunching or just serving files? Can you leverage 64-bit power? Need much more memory per process?

If you are unsure about it, the best thing you can do is go and figure it out! We took ourselves the time and tried various options, conducting a lot of IO- and especially MySQL benchmarks. But benchmarking is a complex matter that deserves an article on its own — hopefully coming up soon on this website.

So what did we finally choose and does it work for us? See yourself:

mp:~$ uname -r ; uptime 2.6.18-4-amd64 19:35:07 up 72 days, 7:37, 1 user, load average: 0.06, 0.08, 0.08

The End is The Beginning is The End*

webfactory GmbH — Thu, 27 Sep 2007 00:00:00 +0200

Web design has come a long way. We strive to play the game in adherence to Standards, we separate structure from presentation as much as possible and we have taken into our hearts the fact that indeed the Experience is the Product. Or have we?

A core ambition of website designers is to keep visitors on a site, get them to consume the content and perhaps even interact with it. Why then are users still left to themselves after they have done what should bring joy to every site owner: after they have read all of the content? Too often they are fobbed off with just a copyright info, a legal notice or a claim to W3C validity at the end of the page. They are met with bland place holders where they should in fact be rewarded for their effort.

Let us imagine a user that has just devoured all of the beautifully laid out information on a page. He is a most wanted user: he made the effort to scroll down all the way. He might have gotten engaged in an interesting train of thought, or become convinced of a certain product’s advantages – in any case, he is entertained, even excited to a point. Now the user has arrived at the bottom of the page and is looking for more to read, to absorb, to engage with. Ideally, we would want him to stay and navigate to another part of the website. Ideally, he would find recommendations where to go next. Unfortunately for both of us, he is usually faced with nothing helpful and left to himself.

Footers are in serious need of some love. We have to pay more attention to what is going on (literally) “down under”. Just like Europeans acknowledge the existence of Australia but rarely know what is going on there at any given time, footers still are a bit of a hazy area in web design.

Carpe Pedem

What are possible solutions then? To be honest, we (as designers) are only bound by our imagination. That is the beauty of it: once footers are understood not as a point of probable departure but as a point of pending decision, the possibilities are endless, as long as they stay in line with the context of the website.

One of the most common uses of the footer area is the placement of a feedback form. Gaining immediate feedback can be immensely valuable, and is most famously used in blogs. Bloggers have almost unanimously included comments at the bottom of every article, but news and corporate sites also are slowly beginning to embrace this opportunity for user feedback.

Other pages, like last.fm or skype.com put almost their complete navigation or sitemaps beneath the content area. Whereas this certainly offers the user ideas about where to go next, the sheer amount of options may cause the opposite effect of what is intended.

E-commerce pages may want to display related products to the one described on the respective page (a feature promoted by Amazon, for example), a link to an order tracking form or to their support area, and possibly customer reviews.

A company providing services might offer links to other services than the one highlighted on the page and showcase excerpts of their portfolio to complement the description of their work.

Footers could also include links to partner organisations, contact information, promotional elements, recent news – again, there are endless possibilities.

Just as long as we comprehend our footers’ potential as beginnings instead of mere endings.

(* Courtesy of Smashing Pumpkins, 1997)

References and Further Reading

"WEB 2.0 Footers - How to keep people’s flow"
"Web Design Elements : Footers"
"Blasting the Myth of the Fold"

Meeting Alistair Cockburn: The Secret of Effective Teams

webfactory GmbH — Sat, 08 Sep 2007 00:00:00 +0200

Alistair Cockburn is a well-known author and consultant on software methodology. I was particularly impressed by his books Surviving Object Oriented Projects and Agile Software Development. Recently, Alistair gave a workshop in Bonn titled “The Secret of Effective Teams – Agility, Discipline, Focus”, so three of webfactory (Per, Matthias and Sebastian) took the chance to meet him and learn about his newest insights.

Alistair started the day with a talk summarizing some of the most important ideas (in my opinion) from his book Agile Software Development, such as

Three stages of learning a skill (Shu-ha-ri), one of my favourites
The cooperative game, swamp game example
Programming as theory building – great paper by Peter Naur (1986), reprinted in the Agile Software Development book
The role of the project manager
Agility and discipline

This part of the day was a nice, relaxed refresh of what I had already learned from the book. Nothing really new so far, but still the ideas are very good and the book is my strong recommendation.

The rest of the day was made up of group assignments. We did the “draw a drawing” communication game (one group has to draw a drawing based on textual instructions from the other group) in an agile version, meaning that there were three drawing rounds with reflection sessions after each round on how to do better next time. Alistair put a special emphasis on “cheating legally”, which means breaking assumed rules (as opposed to real rules) to improve project success. In fact, most of the improvement between the drawing sessions resulted from breaking assumed rules. Also the reflection workshops proved to be an important technique for success, which inspired us to do more of them in upcoming projects. In fact, reflection workshops are a technique we already used at webfactory for past projects, but not as regularly and often as we could have.

There were two more group assignments related to project management (Information Radiators, tracking project process) and concurrent development in the mid afternoon, and a “goldfish bowl” discussion at the end. Goldfish bowl meant there were a maximum of 5 persons discussing at a time, and the rest of the participants listening. Everyone wanting to join the discussion could take the place of one of the people discussing. The goldfish bowl discussion was there on special request of the workshop organizer, as Alistair stressed, and in fact neither he nor the participants had a good idea of what to use this discussion for. So we tried our best finding topics to discuss, but nonetheless the discussion was at no time really vivid. It ended with Alistair and two of webfactory, Per and Matthias, sitting on the discussion chairs, my colleagues trying to get Alistair to talk about agility topics.

To summarize the day: The morning was quite interesting and energetic. Maybe because of the lenghty three-course lunch, the workshop somehow lost drive in the afternoon. Especially after the drawing sessions, one game/group assignment seemed to follow the other and the workshop speeded up noticeably due to the fact that weekend was approaching.

After all, what did we get out of this workshop? Most importantly, the reassuring feeling that we’re on the right track with our approaches to agile software development here at webfactory. It was particularly great to learn that agile development and fixed-price contracts are not only compatible with each other, but that Alistair is even convinced that fixed-price contracts are the only reasonable contract model for software development and – the other way round – that agile development is the best way to cope with fixed-price contracts with tight budgets. This is the opposite of what most speakers on the OOP 2007 conference in January argued for. To us, it seems to be a more realistic view of project reality and encourages us to continue working the way we have always worked.