Why New Technology is so Complicated

14. May, 2009

Ever wondered why the new cool thing is so complicated? There is a very good article which explains just that. In a nutshell: When the technology is invented, it’s invented by experts in the field. They have toyed with this idea for years, refined it, applied it in numerous projects and honed it until something new and useful came out.

Next come the early adoptors which are usually also experts in the field. They are always searching for a new, better solution and they are actively searching. The also have the background to understand what a new technology means for them, since they have the experience.

After that comes the normal user. The normal user has little idea what is going on, she just “wants to solve this simple problem.” The documentation (so far only written by experts for experts) mean little to her since she simply doesn’t have the background. She also doesn’t want to become an expert, this is usually going to be a single-strike project, so there is no intention to spend any time on learning the technology.

4 Comments | Philosophy, Software | Tagged: | Permalink
Posted by digulla

env.js is Back

11. May, 2009

After quite some time of inactivity, env.js is back. There is a Google group and a git repository.

In case you’re wondering what env.js is: It emulates a web browser in pure JavaScript. What on earth could that be good for? This allows you to run your web application in a unit test. You can write your JavaScript, load env.js, your own code and then run it. You’ll have access to document, events, the DOM, everything. No browser bugs, yet, but that will probably come, too. With this gem, you can finally run your web app in a single process, with every bit of information readily available to your IDE’s debugger. No more messing with a remote or local web server, deploying your application and hoping that Tomcat could reload all classes, no more external browser process and guessing what might cause the odd behavior.

Leave a Comment » | Software | Tagged: , | Permalink
Posted by digulla

Why aren’t IDEs intuitive?

8. May, 2009

Mark asked a simple question on stackoverflow.com:

This is stupid. Why can’t an IDE be intuitive enough that you can “good” at it immediately, and “great” after picking up a few shortcuts?

My answer is: Because computers aren’t powerful enough for this, today.

In the most simple case, the IDE would need several ways to layout the data on the screen, it would need several predefined keymaps with shortcuts (because different people do different work in the same IDE) and it would need a way to figure out who is using at right now to switch defaults. You would need to write two times the amount of code to implement all this because some thing are they way they are because the rest of the code is the way it is. IDEA can’t compile in the background, Eclipse can. To allow both ways of working, you would have to rewrite parts of the compiler API, the way the compiler talks to the UI, and in the IDEA case, you’d probably have to change the compiler itself.

Taking this one step further, the IDE would need to learn how the user works, what (s)he cares for, how (s)he thinks. The user is always resizing the console after starting the program? Let’s do it for her. It irritates the user that the IDE hangs for the fraction of a second when the IDE switches to the debug view? Let’s load the debug code in advance in the many spare seconds while we wait for anything to do.

Leave a Comment » | Philosophy, Software | Tagged: | Permalink
Posted by digulla

Trojan on ATMs

2. May, 2009

A few weeks ago, I stumbled over this: It appears that criminals have managed to install a Trojan on Russian ATMs. The Trojan would collect card data and pin numbers over the day and during the night, a “money mule” would collect a receipt with the numbers (which would look inconspicuous since a lot of people ask for a printout of their transaction). But this kind of attack is a new quality.

Home computers are administrated by … well … ignorants. People who want to use a computer, not understand it. For them, this box eats electricity and magically produces fancy graphics on the screen. They know how to email but they have no idea how mail works, they are oblivious to what actually happens when the computer sends an email. So it’s little wonder that most computers out there are infected with various kinds of viruses or Trojans and why “MAKE MONEY FAST” schemes still work so well.

The guys who build ATMs, on the other hand, are no ignorants. They ought to know exactly what they are doing and that someone can tap into the process is a new dimension. This is the difference between mugging innocent night owls and planned bank robbery. Computer crime has become as professional as the non-virtual counterpart. My guess is that we’ll need much more powerful computers in the near future which can store and access petabytes of data. Computers who can tell a legitimate operation from an illegal one and who can protect themselves against abuse. Computers who are powerful enough to watch every operation they are processing. Instead of only being able to crunch numbers, they need to understand what they do and how far reaching the consequences of an operation are.

It’s time for an immune system for computers. If we’re wiped out by Skynet in the not-so distant future, we’ll have to thank the mob.

Leave a Comment » | Software | Tagged: | Permalink
Posted by digulla

PyScan: A Little Helper For The HP CM1312nfi MFP Scanner

18. April, 2009

Update: Find the latest version (0.6) here.

I recently bought a HP CM1312nfi MFP scanner (multi function device with scanner and color laser printer). After scanning some 1’000 pages, I’m still satisfied with the device. The document feeder (ADF) sometimes tries to eat the paper after spitting it out and the colors could be a little more brilliant but overall, a good deal for the price.

What bothered me was that the “Start scan” button doesn’t work on Linux. But someone posted a script in the bug report which can poll the button by reading the URL http://$ip/hp/device/notifications.xml (replace “$ip” with the IP address or DNS name of the scanner). This returns some XML with two interesting elements: StartScan and ADFLoaded. The first one becomes 1 when someone presses the “Start scan” button on the scanner and the second one tells us whether there is some paper in the ADF.

With that and some source code, it was simple to create a little tool that works quite like xsane but fixes a couple of things that annoyed me for a long time:

  1. The UI of xsane is dead while it scans
  2. There is no online preview of the scan; you have to open the file in some extra tool to verify that the scan looks OK
  3. xsane doesn’t know about scan “projects”
  4. xsane doesn’t start to scan when I press the button on the scanner

As with all OSS software, this thing can seriously ruin your day, so be a bit careful. One of the biggest problems is the file size: To be able to edit files without loss of quality, TIFF format is the default. Each full page scan takes 26MB, 100 pages need 2.6GB!

Plans for 0.5: Allow to edit projects in the UI, select them, save and load them. Right now, you must define your projects via the command line or by editing the source code.

Download: PyScan-0.4.tar.gz (12KB, MD5 checksum)

Dependencies (see README.txt for download links):

  • Python 2.6
  • PyQt4 4.4.3
  • Python Imaging Library 1.1.6
  • Python Imaging SANE 1.1.6 (needs included patch; see README.txt for instructions).

Features

  • Code to load images in a background thread, generate thumbnails (compatible to Konqueror/Dolphin) and display them in a list view
  • Display a (big) image with various manual and automatic zoom levels and modes (fit to window, percent) with zoom and pan
  • Online preview of scan in progress

Hideous details of the source

Again and again, I’m astonished how simple some tasks are in Python and Qt … if you’re willing to accept some “non-OO-ness” of the solution. I’ll explain some things I did here to give you an idea what’s going on.

Online preview

PyScan has an online preview of the currently active scan. If you look at the documentation, the Python Imaging SANE interface offers no way to do that. After looking at the source, I found that the SANE interface simply reads bytes from the SANE scanner module and copies these into a PIL image which was created on the Python side.

So my solution is to be notified that a scan is in progress and then copy said image every second (all 26MB) into a string. That string is then used to build a QImage which is turned into a QPixmap which is then displayed in the right pixmap view. See pilImage2QImage() for the details.

Background threads

I also moved all expensive code into threads: Loading big TIFF images, scaling them down to thumbnails, saving the images, etc. All threads have a method to add work to their input queues and they send Qt signals when they’re done. Continuing the scanning when there is paper in the ADF tray was a bit of a problem, through.

Since the saving of the images is happening in a background thread, the code could start the next scan before the saving was completed. This wasn’t such a big problem except that the “scan next image” code looks for files on the disk to determine the next filename. This would lead to overwrites. So I had to synchronize this somehow. My simple hack was to set a boolean “waiting” in the scanner thread which indicates that the scanner has more paper to process and waits for the save thread to complete. When the UI gets the “image saved” signal, it triggers the scanner to continue.

Generating thumbnails

The last hack in the code is the generation of the thumbnails. The main issue here was that I need the thumbnails for the gallery view really deep down in the Qt render code. Wasting time at that level is really a no-no but at first glance, the API offers no way to defer loading of the images and then later update the items in the list view when the data is available. Keep in mind what I need to do:

  1. Load a 26MB file from disk
  2. Scale it with antialiasing
  3. … for hundreds, possibly thousands of files!

My solution: In the render code, I create a LazyPixmap. This is just dumb object to save the filename and a placeholder pixmap which is used into the real thumbnail becomes available. The LazyPixmap will schedule a job for the LoaderThread.

In my first code, I tried to create a QPixmap in the LoaderThread but that doesn’t work: Only the UI thread is allowed to create a QPixmap. Duh. But luckily, Qt offers the QImage class which works even without a UI and which offers basically the same API as QPixmap. So the LoaderThread can load the image from disk and scale it down (to save memory and avoid heavy computation in the UI thread) right before emitting a “loaded” signal.

There are two places where a LazyPixmap is used: In the PixmapWidget (which can display and zoom a QPixmap) and in the ThumbnailDelegate which draws the thumbnails for the filenames in the GalleryModel.

In the case of the PixmapWidget, the signal will be handled in lazyLoaded(). Here, we convert the QImage into a QPixmap (in lpm.getPixmap()) and assign that pixmap, recalculate the zoom factor, realign the view, etc.

The GalleryModel, I have the problem that I need to tell Qt somehow that the pixmap has changed but the API offers nothing except rendering the whole widget by calling update(). This will render at most (on a huge screen) 30 pixmaps. Happens one time per visible pixmap, causes no flicker. Probably not worth to waste another second on it.

If you look at the code, you’ll see that a class called KDEThumbnailCache is used. This class accesses the same thumbnails als konqueror (KDE3) or Dolphin (KDE4). This means once the images are scaled down (either by my code or Dolphin), all tools can quickly load the small, precalculated thumbnails instead of having to scale the 26MB files again.

Conclusion

Well, that’s it for a small walk through the code. Feel free to give feedback if you like PyScan (or not) or when you have patches.

5 Comments | Software | Tagged: , , , , , , , | Permalink
Posted by digulla

Testing the Impossible: Asserting Several Values At Once

23. March, 2009

So you want to check several values at once, maybe because that helps you to locate the error more easily. Simple:

int v1 = list1.size();
int v2 = list2.size();
int v3 = list3.size();

assertEquals (
      "list1=5\n"
    + "list2=2\n"
    + "list3=0\n"
    , "list1="+v1+"\n"
    + "list2="+v2+"\n"
    + "list3="+v3+"\n"

2 Comments | Software | Tagged: , | Permalink
Posted by digulla

hpaio: unable to read device-id

21. March, 2009

I recently bought a multi-function device printer scanner (HP Color LaserJet CM1312nfi MFP). Setup on openSUSE 11.1 was pretty painless with the hplip package (version 3.9.2) from packman. The original openSUSE package would core dump. Duh.

Today, I had a strange error all of a sudden:

scan/sane/soapht.c 486: unable to open device hp:/net/HP_Color_LaserJet_CM1312nfi_MFP?ip=xxx.xxx.xxx.xxx
io/hpmud/jd.c 84: unable to read device-id

Of course, I hadn’t changed anything in my setup … until I remembered that, for security reasons, I had disabled SMNP in the web interface of the printer. Guess what, after enabling “SNMP-Lese-Schreib-Zugriff aktivieren” (“Activate SNMP-Read-Write access”), the error went away.

For the record: Most functions of the device are accessible from Linux:

  • Color printing
  • Scanner
  • Automatic document feeder (ADF)
  • Remote printing and scanning

I haven’t yet managed to make the “Scan to PC” button on the front of the device work but that’s just a minor issue for me.

The price/performance ratio is very good. The printouts look very good and the ADF could cope with all media that I’ve fed him so far (even these thin, glossy, cohesive catalogue pages; you just have to spoon feed them one at a time).

The scanner is good, but not top notch. The quality of the scans is OK, colors are pretty close to the original but the images could be sharper. The scanner uses a CCD line, so no warmup time (if you don’t have to switch on the whole machine).

If you want better quality color copies, set the resolution to 300 DPI (factory setting is a poor 150 DPI which makes thin lines really smear).

1 Comment | Software | Tagged: , , , , | Permalink
Posted by digulla

Random Idea: Public Data Safe

5. March, 2009

A lot of places offer free data storage these days but I’m concerned about security. How safe it is to store something sensitive there? You may think that your address is not sensitive data but you’ll have to reconsider as soon as some crook uses it to open a new eBay account…

So my idea is create a public data safe where you have full control what someone else can see. For example your address: Instead of giving a online shop your address, you give them a random ID and you give access rights for the delivery service to resolve that ID. The shop doesn’t know where you live (and your address doesn’t go “somewhere” when the shop goes bankrupt) and you still get your goods.

Imagine you have an accident. You’re unconscious and are brought into a hospital. They need to run tests to figure out your blood group. If they have to give you medicine, then you’re better not allergic to it. And how about alerting your relatives? Here, a random ID which only registered health care services can decode would help: You could put your health data (allergies, medicine that works better on you than other, blood group, relatives to alert, important facts like whether you have a heart disease, etc) online without risking to find them on the loose.

How would it work? The service itself would only store data which you send encrypted. So the encryption would happen on your computer. Better get that anti-virus up to date, though. Health or delivery services would put their “public keys” online on this site, too, so you could encrypt the data in such a way that you and they can read it (basically, you make a copy and encrypt both). This way only certain people can read the data.

When data is uploaded, it gets tagged as “address”, “name”, “age”, “medical information”, etc. and each piece gets a unique, random ID (so you can’t tell from the ID who it belongs to). When another service should be allowed to see some of your data, you take their key to encrypt a copy for them and send them the ID.

If your address changes (because you move), you’d have to update it only in a single place.

Drop me a comment if you like to implement this idea.

2 Comments | Software | Tagged: | Permalink
Posted by digulla

Random Idea: Guided Public Transport

5. March, 2009

Just got this idea: You can download train and bus schedules on your mobile, you can buy tickets online, so why doesn’t the system combine the two to guide me through the public transport?

The idea is that I send a request to travel to some spot. The software should know which coupons and other discounts apply for me, what tickets I already own, etc. So the first step would be to tell me what tickets I need in addition to what I already have and what that would cost.

If I pay, the system should guide me to the next bus stop or train station where I can start my journey. During the travel, the system should inform me a few minutes before arrivial at my next destination. This should include the estimated time of arrival, where I’ll arrive (which platform), where I need to go next (platform or location on a map), how much time I have and what connections are available.

It should also be possible to interrupt a travel, say, to have lunch on a long trip. As necessary, the system should buy new tickets and cancel preordered ones. If I stay overnight, I should get a list of connections on the next morning so I can ask the hotel service to wake me at the right time.

If you like the idea and want to implement it, drop me a note.

Leave a Comment » | Philosophy, Software | Tagged: | Permalink
Posted by digulla

Verifying Results in Tests

5. March, 2009

So you finally got yourself writing a test against a database. In the test, you have to verify that a row in the table is correct, so you write:

assertEquals (
    "2008-09-16-13.50.18.000000;;;;1;2008-08-07;2008-08-07;JUNIT;2008-09-16;t0001;001;;Doe;Jane;;Street;2;Doe Jane;;;;;X;2575;John;;;;US;E;;01;;;;125;01425;0;Shop;;;DOE;JANE;JOHN;032;1;;0001010301;;;;",
    dumpRow(key));

and it sucks. Yeah, junit will notify you when something in that row is wrong and if you have a cool IDE, you can even compare the fields … but it still sucks. If one of the fields in the middle change, you have to scroll and eyeball-diff, wasting your time. The solution is pretty simple:

assertEquals (
    "2008-09-16-13.50.18.000000\n"
    + "\n"
    + "\n"
    + "\n"
    + "1\n"
    + "2008-08-07\n"
    + "2008-08-07\n"
    + "JUNIT\n"
    + "2008-09-16\n"
...
    dumpRow(key).replaceAll(";", "\n");

Instead of dumping the data in a single long string, split it into lines so you can compare fields side by side and without scrolling sideways.

Leave a Comment » | Software | Tagged: , , , | Permalink
Posted by digulla

« Previous Entries
Next Entries »