It’s fair to say that Australia’s Attorney-General, George Brandis, has been struggling to explain what he means by metadata:

Brandis: “The web address, um, is part of the metadata.”

Journalist: “The website?”

Brandis: “The well, the web address, the electronic address of the website. What the security agencies want to know, to be retained is the, is the electronic address of the website that the web user is … “

Journalist: “So it does tell you the website?”

Brandis: “Well, it, it tells you the address of the website.”

It appears that Brandis is trying to explain the difference between the address of the webpage (the URL) and the numerical address of the computers serving up the webpage and accessing the webpage (the IP addresses).

If so, then there are a couple of problems with the Government’s proposal:

  • a single IP address may serve web pages for many different and unrelated websites.
  • a home user may have many different IP addresses over time

For example, this webpage is hosted by WordPress, as are many many other webpages.  WordPress uses the URL to know which website to serve, not the IP addresses.  Indeed, WordPress use many different computers with many different IP addresses.

Furthermore, and there are exceptions to this, but most home users, when they connect to the internet, will be given a different IP address each time. There are a finite number of IP addresses, and this may change in the future, but at the moment, common current practice is for ISPs to share IP addresses between customers.

As described, the government’s proposal will not work.  To know who is visiting what website, more than just IP addresses are required.

I attended a Sydney Tester’s meetup yesterday: Help … Can you solve my testing problem?

Three attendees volunteered problems that were discussed:

  1. What is the role of a tester on an agile team?
  2. How to retain motivation in a challenging environment?
  3. How to test in a multi-tier environment?

1. what is the role of a tester on an agile team?

  • key issue: even on an agile project, management still need to be able to track (at least approximately) the percentage complete/to-go for the current sprint
  • discussed: ways of logging exploratory tests on the fly – for audit purposes, and as potential input into formal test cases
  • identified: test phase squeeze is an issue – coders overrun schedule => testers get squeezed
  • suggestions: involve testers earlier – when specifying user stories; move some testing responsibility to coders; various ways to predict effort required.
  • open issue: how much documentation is ‘sufficient’?
  • open issue: test automation should save time, but how to find the time to automate tests?

2. how to retain motivation in a challenging environment?

  • investigated: if overwork is the issue – concluded no
  • workplace specific issues: inexperienced developers
  • key issue: determined that test fail rate is very high – approx 75% of tests failing – led to discussion of ways to ‘train’ developers, also led to speculation that tester may be aiming for a higher level of quality than customer actually wants

3. how to test in a multi-tier environment?

  • key issue: tests depend on data which is pulled from/pushed to a chain of other systems which are not under projects control
  • noted the risk that available data will drive test selection instead of the other way around
  • suggestions: automated searching of data set for usable data, mocking data in database, mocking data by capture and replay at external interface.

Interestingly, at least two of these problems are people problems, not technical problems. The solution will lie in realigning someone’s expectations – perhaps everyone’s.

Is Agile Fragile?

This was the proposal put to me several times at iqnite. It contains a grain of truth. It also represents a misunderstanding of what Agile is.

Obviously, there are many ways in which any process can fail. The question I want to address here is Agile’s reliance on automated testing.

Major changes in direction, Refactoring, Collective Code Ownership, Short Iterations: all of these things require significant testing – either manual or automated. Except of course, manual testing is too expensive to be done often. With manual testing, you are locked into long iterations – unless you are prepared to deploy untested code.

But automated testing is no panacea.

Too many automated test-suites are clogged with broken tests, tests that cannot be run, tests that fail but are not fixed.

Broken tests are not the same thing as failed tests – the very purpose of any test is to fail when appropriate. A test that passes is broken if it won’t fail when it should. A failed test is a not a broken test if it leads to a code fix.

Developers rarely commit failing tests. Tests fail when things change.

Developers frequently commit broken tests. Tests break when a developer assumption is violated or incorrect.

Developers necessarily make assumptions about the test environment, business rules and the behaviour of the rest of the system. They have to. Testing is the art of capturing assumptions. A test is an assumption captured in a testable format: “Test that when X happens, the system does Y“.

But there are better and worse ways to approach capturing assumptions.

The standard lazy way to capture assumptions is to manually load test-data into the test-database, create your code, observe the output and then create tests to validate that the code reliably produces that output. The best that can be said about such tests is that they are cheap to create – they can be created even when the coder does not know what output the code would create or should create.

The downside is that they are fragile – they are highly likely to be irreparably broken, even though they pass. They often fail when they shouldn’t and frequently don’t fail when they should. When such tests fail, they rarely point clearly at the violated assumption. Instead of analysing the target system, the tester all too often spends time debugging the tests,sometimes just to get them to run. Even when they are ‘fixed’, they are still prone to break again the next time anything in the system changes. The cost of maintaining badly written automated tests very often exceeds their ongoing value.

The better way to capture assumptions is to have the test-suite load the necessary test-data into the test-database, to create one test at a time, to create each test before creating the code that makes that test pass, to make each test as small as possible, and to make each test pass before moving onto the next test. This is harder, much harder, because it requires the developer to have a fuller understanding in advance, rather than just trying and tweaking until it ‘looks’ right. But it produces more robust and reliable tests.

Having the test suite create the test-data eliminates the risk that the data will not be there – and eliminates the need for future testers to reverse engineer the data the test needs.

Implementing tests one at a time, and keeping each test as small as possible, reduces duplication between tests and reduces the risk of creating untested code. This reducing the rework required when business rules change because it reduces the number of ‘false positives’ and ‘false negatives’; it reduces the risk that a single change will cause many tests to fail, or even worse, not cause any tests to fail.

Implementing the test before the code means that the test validates not just the code but also the business rule; it validates the what the code should do, not just what the code does. This reduces the risk of creating a test passing only when the code does the wrong thing.

As with code, the higher the initial quality, the lower the ongoing maintenance cost. The cost of maintaining well written automated tests is usually manageable.

Tests that signal only that something has changed and that debugging is required can have a place, but they are really a form of manual test. They need to be updated every time anything changes, and they need a human to interpret the result.

Manual tests will always be more robust. They require less effort to update, because they are less explicit, relying on the tester for the extra details. In manual testing, the tester can be expected to interpret vague instructions, to correct minor errors on the fly, to adjust for changes in business rules, and to intelligently evaluate variations from the originally specified results as either pass or fail. All of these things break automated tests. But of course, manual tests are more expensive to run – at least when compared to well crafted automated tests.

Automated tests will never be as robust, but that does not mean that they have to be brittle, if sufficient care is put into their creation. They are more expensive to create and maintain than manual tests, but can have a lower total cost because of the cost of running manual tests, and automated tests have intangible benefits: they are repeatable, they can test things that are not accessible to manual tests, and they can be run more often and in full. And how best to take advantage of those benefits is a topic for another blog post.

Comparing OSDC  (the Open Source Developers Conference) with iqnite (the conference for professional software testers), if you guessed that OSDC would be ‘cooler’, more hard-core and more feral than iqnite, you’d be right.

If you also guessed that OSDC would be less concerned with matters financial, you’d be wrong. Money was explicitly discussed far more often at OSDC than at iqnite.

To understand why, remember that the typical iqnite attendee is a professional at a VBC (Very Big Company), quite likely to be concerned with the effort estimates, schedules, and targets, all of which have dollar implications, but only indirectly. Further, that concern is usually local, often not going beyond the person’s own contribution. As the old saying has it, “The operation was successful, but the patient died”. Or, as a salesman once said to me “I don’t know if the projects I sell make a profit. That’s not my job.”

On the other hand, a significant proportion of OSDC attendees are small business owners, precisely because they have a passion for technology but – ironically – to survive as a small business person requires a constant focus on the financials. Or they are running, and funding, Open Source Projects.  Therefore, there were topics such as: “How to sell, for geeks” (Giri Fox), “Small Business Mistakes” (Jacinta Richardson), “Free as in Kittens v2.0″ (Evan Leybourn) and “Blood from Stones: Asking for money for your project (Cat Allman).

A theme common to both conferences was the move away from bespoke development from scratch, towards building on (integrating) off the shelf packages. Paradoxically, as the utility of 3rd party platforms increases, so does the number; it becomes simultaneously more necessary and more difficult to keep track of new developments in any given field.

And of course, there were robots. There was a 3D Printer. There was a mind-controlled mouse. As one speaker put it, “the future is here”; all of these things which used to be science fiction are now affordable by hobbyists and will eventually be every-day items.  If you were at OSDC 2012, you saw it first.

One more thing. There has been a turnover of the OSDC executive committee – so expect some changes and improvement for OSDC 2013.

No scripts in this post, just a quick review of what was topical at iqnite 2012, the software quality and testing conference. Read the rest of this entry »

For the testing of servers

If the only tool you have for testing your server is a GUI, you are going to be embarrassed occasionally. Read the rest of this entry »

A simple calculator

This is a simple calculator that sits in the gap between feature-light programs like windows calculator and the high overhead of a full on spreadsheet. Read the rest of this entry »

For the testing of clients

When developing a client application, it helps to be able to test it.  Sometimes, testing against a real server is enough.  But at other times using a real server will slow you down, either because a real server isn’t available, or because you have more coders than real servers to test against, or because it takes time to set up the server for each run, or simply because you want to test behaviour a real server doesn’t normally display, such as error conditions.

In such cases and many more, you need a mock server.

Enter mock_server.pl. Read the rest of this entry »

Wherein we debug what we started  earlier.

If you ran the example at the end of the post tap.pl you would have seen a blank screen.  Excitement.

But why?

Read the rest of this entry »

Monitoring communication between a client and a server

Tap.pl is one of the oldest scripts I’ve written that I still use, and I still use it regularly.  Some of the other scripts I will share depend on this one, so it seems like a good place to start.

At its heart, it is incredibly simple: Receive a message, keep a copy and pass it on. Read the rest of this entry »