ibm_champion

IBM Champion!

IBM announced the nominations for IBM Champions 2017 and I’m very happy to say that I was nominated.

It’s wonderful to have this sort of recognition and suddenly being part of a group who have made huge contributions to the IBM ICS Community of the years is humbling.

I strongly suspect that my nomination is linked to my co-organising the Swiss Notes User Group, so I’d like to mention at this point my co-conspirators, without whose work SNoUG wouldn’t happen:

Diana Leuenberger, Susanne Schugg and Helmut Sproll

And also many thanks to those that have contributed behind the scenes to make SNoUG a success – I think in particular here of Paul Harrison, with his wonderful AgendaWALL and AgendaPAD, as well as the very talented LDC Via Group, who help me whenever I’m stuck.

I’d also proud of strengthening the Swiss IBM Champions group – showing that plucky little Switzerland can pack a good punch!

screen-shot-2016-11-30-at-16-58-34

From 3000 milliseconds to 200 to open a page: Speed up your Notes Client applications by discovering their bottlenecks.

I had inherited a template for a database which was taking 3 seconds to open any sort of document. It soon got to be irritating, and I noticed that the little lightning bolt at the bottom left side of the Notes client was flashing wildly, so I fired up the Notes RPC Parser to try and see what was all the traffic about.

The culprit was soon found: it was a subform containing buttons whose label was dynamically computed depending on the user’s desired language. And, unfortunately, it was making an @DbLookup for every single button, and an uncached one at that. If you look at the screenshot, you’ll see that an @DbLookup generates 3 NRPC Calls, for a total of 50ms – and this was done 30 times each time a document was opened.

That, and another issue that I corrected, meant that I could massively improve the performance of the application for only about half a day’s worth of work.

screen-shot-2016-11-30-at-16-58-34

Feel free to use the tool, or if you can’t be bothered, hire me to solve your Notes Client performance issues.

P.S. I’ve just released a new version of the tool Notes RPC Parser, which you can download here:

ldcvia

Host your migrated Domino Data in Switzerland at ch.ldcvia.com

LDC Via is a dream destination for your Domino data. If you are looking to decommission your Domino infrastructure, LDC Via offers you a Platform-as-a-Service (PaaS) solution for hosting your migrated Domino data. Once the data are transferred, they are available using the very modern REST interface which all modern Web applications (including native mobile applications) can easily access.

Certain Swiss customers (such as local governments) are not allowed to host their data outside of Switzerland.

For these customers, and those of you who wish to avoid any interaction with US companies because of legal concerns, we have set up a Swiss instance of LDC Via, which is immediately available under

ch.ldcvia.com

I invite you to start exploring the service! Please contact me at andrew.magerman@magerman.com or +41 79 831 83 13 if you have questions about the service.

The pricing for the Swiss service follows the tiers defined at ldcvia.com/prices.html

Category monthly price
Basic Free
Standard 400 CHF per month
Pro 1’000 CHF per month
Enterprise 2’000 CHF per month

all prices excluding VAT.

Why do Software Developers always squawk when inheriting a foreign codebase?

The first thing that Software Engineers do when they inherit a codebase that they haven’t written is to squawk loudly.

This is terrible! That’s not the way to do it! We need to re-write this completely!

I recently had an experience when a project manager suddenly lost his developer and asked me to take over. ‘Here is the list of requirements from the customer, they’re just small changes’.

I said ‘OK, I’ll give this a look, doesn’t look too difficult‘. And then I started looking into the code, and I started grumbling more and more. There were no comments. No source control. No test code. The code I had in front of me what not working correctly, and I had no access to a codebase which was working. And within the code, there were so many times when I was thinking ‘Oh no, why would you do that? I’d really do it differently’.

So I promptly squawked. And the project manager expressed his incredulity. ‘It’s always the same with developers! They always complain of code that they haven’t written themselves!’.

So, here’s a little guide to explain why this happens.

We’re not masons.

That’s the crux. The job of building walls is well defined, and if a wall is not complete, you can hire another mason to finish the job. He’ll come to the site and the tools that he sees are familiar and he can start right where the previous person stopped. Easy.

That’s not how it works with software. There is a wide range of ways you can solve something, the wider since we basically can write your own tools. Another persons’ building site is going to be completely unfamiliar.

So here are my insights for non-developers, in the hope that you will get over the impression that we are whiners, and understand that software development is not masonry.

Software is easier to write than to read.

After about six months, even the code you have written yourself becomes more difficult to read. A good developer will be writing his code not just well enough for the machine to understand (bad developers stop at that point), but well enough for humans to understand.

This is a pretty accurate picture of what it’s like reading other people’s code from somebody else:

Software can be really badly written, and you can’t see it from outside.

This rejoins the points I was making in a previous post. Here we’re talking about maintainability, the ease with which a particular codebase can be modified.  Here are some best practices which you should insist that your developers follow:

  • One is to separate the responsibility of code into little independent boxes, which only do one specific thing. If you need to change something, there is only one place to look for. This is called modularity and follows the concept of separation of concern.
  • Another one is having Source Control. This shows all the changes that have been done on the codebase over time, including who did it, and enables us to revert code to what it was in the past (when a bug has been inserted, for instance.)
  • Automated tests make every small bit of the code run through its logic, often by trying to make it break. There is one thing that we software developers are scared of, and which happens a lot: we change a bit of code here and then notice that loads of other bits of code are no longer working. Automated tests enable you to be able to make bold changes because you immediately get feedback on whether you’ve broken something.
  • Finally, documentation really helps. And please note that I’m not talking about some separate Word Document that was once done for the client, I’m talking about comments explaining why things were done in a particular way embedded within the code.

If a software developer inherits a codebase which didn’t follow these guidelines, then that code is going to be a pain to maintain and to change. And as a developer, it’s a large source of frustration because we are in effect far slower at delivering changes than our predecessor, and our customer is not going to be impressed.

So, if you are a customer, request that these basics are followed, or else you’ve got a bad case of developer lock-in, where Bob is the only person who can maintain the code.

In conclusion:

If you are a customer and switch developers or provider for a project, you’ll have to factor in some budget and time just for the new developer to get acquainted with the project. You’ll also see that it will initially take longer to effect changes, depending on how well and maintainable the software was written in the first place.

 

ldcvia

LDC Via is a dream destination for migrated Notes applications

It’s becoming increasingly clear that IBM Domino/Notes has reached its software end-of-life.

IBM has not announced it, but the proof is in the pudding, and the pudding says no new releases since 3 years, the pudding says minimal IBM resources earmarked for development (almost all the exciting new stuff has been a community open-source work, many thanks to the heroes at OpenNTF), the pudding says outdated development tools, circa 8 years old. I’m writing this article on the unconfirmed assumption that IBM has no interest in preserving either the Notes Client, the Domino server, and the nsf storage file.

It’s time to start thinking about exit strategies.

For companies that decided to move off from Notes/Domino, after the easy part of migrating mail-files and changing mail servers, they were usually left with a plethora of functioning, sometimes business critical applications, for whom the cost of a complete re-write in another platform was prohibitive (more on that later – it’s still the case). The exit costs are high, and most organisations decided to keep their Notes infrastructure running, in some sort of compromise co-existence scenario, often with the fat clients hidden away or available only to a small subset of users.

But now that IBM is (apparently) pulling the plug, the ‘application migration’ question comes up again. The costs are still high. Sorry. Why?

The ugly secrets of most of the Notes applications that I’ve run across are

  1. They are badly documented (a crime which presumably is prevalent in most in-house developments, as opposed to professional software houses)
  2. The development tools invited developers to work in a quick-and-dirty way, with no structure, producing fast initial results but subsequently slow updates (This was unfortunately also the case for XPages applications).
  3. The absence of any way to do systematic testing also led to an accretive style of coding, where one only adds new stuff, and getting rid of old stuff is so dangerous that nobody dared to do it. This also means that, in a historically grown application, there is going to be a lot of cruft within the application – it’s impossible for a developer to know which are the bits that are needed, and which are obsolete.

You’ll need the following for a successful application re-write:

  1. An application owner from the business side who has followed the history of the tool and will be able to tell the developer which bits are used and which are obsolete
  2. A developer that can ‘read’ the old Notes application, not only in terms of code but also in terms of data structure and security.

Once you have these two sine qua nons, the question arises as to which destination should you choose. Of all the options, my vote goes to LDC Via.

LDC Via makes the re-write job a lot easier. Why?

  1. The data structure is the same, so the accessing logic in the new application doesn’t have to change.
  2. The document-level security upon which most Notes applications run is still available.
  3. The data is accessible using a modern interface called REST which is modern, simple to understand, use and test.

This means that you can really concentrate on business logic and user interface and don’t have to spend time translating the original ‘Notes’ logic.

I’m looking forwards to my first application migration projects working with LDC Via, and I’m also proud to announce that my company has become a technology partner of LDC Via.

Please feel free to contact me if you are interested.

iceberg

Software is an iceberg. You’ll get hurt by the submerged parts.

Software Quality is an iceberg.

The tip of the iceberg is the small part sticking out that everybody can see. A non-IT person judges software with a far smaller subset of criteria than an IT professional: Functionality, performance, and usability.

To a normal user, the interface is the software. I regularly have to choke off exasperated comments when a customer tell me ‘but it’s only one extra button‘.

Lurking beneath the murky waters are those criteria which only IT professionals see: reliability, maintainability, security.

End-users should not be expected to see these ‘underlying’ aspects, but if the decision-makers (i.e. the ones with the purse string) don’t see under the water, you’re doomed to produce bad software. Tell-tale signs are:

  • There is no budget for IT maintenance
  • The only time fresh money is given is for new functionalities – money is never given for a functionality-free maintenance
  • Projects get stopped as soon as the software is barely out of beta testing. Finally, as a programmer, you’ve produced software that works, but in your mind, this is just a first version. There are lots of bits and bobs of useless code left in your software, debug code that writes to system.out, the object and data structure is a mess and could be optimised, you’ve yet to apply a profiler to your code to see where are the bottlenecks. And then the project gets stopped: ‘it works well enough’. That is very upsetting, and possibly one of the main reasons that in-house IT jobs are far less attractive than IT jobs within software companies.

If you recognise yourself as a company, then you should give more power to the IT department. Too many companies treat IT as a necessary, needlessly costly evil. IT should be an essential, central part of your company strategy. There is no other region of a company’s activity which gives you more of a competitive edge.

Too many companies (and I am looking at banks and insurances in particular) have failed to see that they are, essentially, a software company. You need to have a CIO who understands his trade, and accept that he is going to spend money for stuff which doesn’t make sense to you, or at least doesn’t seem to change anything.

If you are only ready to pay for the IT stuff which is visible – more functionality, for instance, but no money for a refactoring exercise which would add stability but not change the functionality, the accumulated technical debt is going to bite you at some point. It might be that an application gets exponentially difficult to change, or it might be that you have a massive data loss that you can’t recover from because the backup software was not up-to-date and nobody did an actual restore as part of regular maintenance.

If you are a developer, and you find yourself in such an environment, you’ll have to go guerilla. You’ll need to surreptitiously improve your code whenever you have the opportunity. Try to leave any code you touch a little bit better than before. Notice duplications and remove them. If you didn’t understand part of a code, then document it after you have understood it. If variable names are unclear, then rename them to something better. It’s an thankless occupation, because there is the risk that you break things that were working, and nobody will really care that you reduced the codebase by half, but it will help you progress and it will increase the quality of your work.

IT is surprising in that often the perception of quality is completely at odds with real quality, because of this iceberg effect. I once used to work in an environment where colleagues were doing terrible things – changing code directly in the production environment, and only doing quick-and-dirty cosmetic patches, but were actually perceived by the customers as delivering excellent quality, because the response times were so fast.

getting git

git is wonderful. In the same way that unix is an’operating system done right’, git is ‘source control done right’.

It has a surprising history, being basically the creation of Linus Torvalds, who decided that all the available source control solutions were useless, and so just wrote his own. That is fabulously cool.

This is a small article to introduce you to git.

Learning Curve

There definitely is a learning curve with git. At least that was the case for me.

For starters, I had to throw away some concepts which I thought were set in stone. Git forces you to think about code in a different way. It turns out that that way of thinking has large advantages.

The single biggest mental stumbling block for me was that I always considered the real code, the precious bit, to be the one I’m currently working on, and the code that is in source control system to be a backup. A byproduct, so to speak. You need to completely overturn this with git.

The real stuff, the precious stuff, is what is in your repository. What you are currently working with, your working directory, that’s something like a scrapbook. You’re trying things out, making changes here and there, but it’s all very temporary and it’s not where the real stuff is. It shouldn’t be frightening to overwrite your working directory, for instance.

Repository

Let me track back a while and introduce the three areas in a local git repository (I am not talking about a remote repository, I am talking about a local git repository, on your local hard drive, and blazingly fast).

The repository is where the history of your project is stored. It’s quite similar to a series of snapshots of your whole project. Each of these snapshots contain your whole project (technically, git is not saving the whole project but just saving the delta to your initial commit, but conceptually, think of these snapshots as whole copies of your project).

These snapshots are called commits. They are unique, and can be referred to in a number of different ways, either directly (by using its number, which is a SHA-1 hash) or relatively (three commits before the last one I did).

Each commit points to a parent commit. When you have two different commits pointing to the same parent, these two commits are the starting points of different branches. The two codestreams, from this branching point, starts having a life of their own, although it’s possible to merge different branches together at a later stage.

Decentralized weirdness

This is a good time to introduce another weirdness that took me some time to grok. There is no primacy between branches. It’s a delightfully anarchic system, no bosses – a little like the anarcho-syndicalist communes of ‘The Holy Grail’. There is no master control program, no centralized master system. One can define certain branches to be more ‘important’ than others, but it’s just a naming convention, it’s not something that is built in.

The same applies between your local repository and the remote repository. Git really doesn’t assume that one of them is more important than the other. It’s very disquieting, but liberating once you’ve understood it.

Working Directory

Next, your working directory. This is where your whole project is, where the code is, images, what have you. This is what gets ‘snapshotted’ each time one makes a commit. It could be possible to snapshot your whole project every time, i.e. every single file, but it turns out that that is a bad idea, because you end up saving too much and it gets really difficult to understand what exactly happened between two commits. I must admit to having done that at the beginning, because I was blocked in my ‘backup’ concept.

No, what you want to do is make lots of small commits, with each commit changing as few files as possible, as long as they pertain to one common change.

Staging Area

Enter the staging area. Really well named, this is where you determine which files you want to belong to the next commit. You’re not interested in the files that have changed only because something trivial like a timestamp has changed, and you’re not interested in unrelated changes either. You really want to encapsulate all the changes necessary for a single debugging or a single new feature into a commit.

Overview

The following diagram shows the basic logic: you add files to the staging area with the git add command, then you commit those added files with a git commit command. Bringing back a specific commit to the working directory is done with the git checkout command.

sublime_text_graphviz_C8OsfB

So, you’ve got this conceptually? Working directory, repository, staging area? Right. This is what you should work with.

I got very confused by how this is actually built. The repository and the staging area are stored in a folder called .git which is contained within the working directory.

This made my head wobble. Surely, not inside the directory? Surely, that way lies folly? Infinite recursions? Surely it should be external? But that’s how it’s built. Inside. It’s elegant once you start thinking about it but it made my head hurt.

Start with the command line

I would recommend staring with a command-line interface. I know that there are many GUI Interfaces out there (SourceTree being a particularly good one), but all they are doing is adding an interface on top of the command-line instructions.

I would recommend moving to a GUI once you’ve learnt the basics of git. Additionally, the command line is very helpful for noobs, as it notices when you have typed something that doesn’t make sense and makes a suggestion. Stackoverflow and the git documentation are the place to go if you get stuck.

http://gitref.org/ is a good, compact resource.

You can download the git command-line bash here.

Recommended Reading

This is a tip from Eric McCormick.

gitinpractise

Git in Practice [With eBook]

Useful Commands

Here I have listed those commands which I am using regularly and feel comfortable with:

Initializing

git init

First initialization, creates the magical hidden folder .git. This is where the actual repository is stored (see above). You’ll need to select the directory which you want to ‘track’

Staging

Staging means ‘saying these are the files I wish to commit’

The command for staging is

git add <filename>

alternatively you can use

git add .

which will add all the files in the current directory (. is the current directory)

Committing

git commit --message='Committing message'

The convention is to write a message with multiple lines, a bit structured like an e-mail. The first line should be like the subject of the e-mail, the other lines are the ‘body’ of the e-mail.

Creating a remote repository:

git remote

shows the remote repository linked to the current repository. You’ll need to have created this beforehand on the github site.

I use github, so the example here is with github. First log into GitHub and create a repository there. Note the URL of the remote repository (there is a convenient button for this)

git remote add origin <your remote repository URL>;

Note: origin here is one of these conventions, it’s the default name of the remote repository. Don’t make the mistake of naming one of your remote branches origin. Remember, convention!

git remote -v

shows the urls linked to your remote repository.

git remote show origin

This command shows all the branches on the origin repository (the remote one, origin is its name by default)

 

 

Fetching

git fetch

fetches the data from the external repository but does not merge the data.

Pulling

git pull

fetches, then merges the data.

Pushing

git push <remote> <local branch name>:<remote branch to push into>

pushes the data from the local repository to the remote one.

Checkout (restore files or change branch)

git checkout

This switches branches or restores working tree files. It’s a bit of an uncomfortable command at first because it seems to be doing two completely different things. Actually, it’s doing the exact same thing – it’s overwriting your working directory with information from a particular commit. Your working directory is a scrapbook, remember?

I find this useful to avoid committing fluff (my technical term for files which have, technically, changed (a timestamp for instance) but which are not actually changes.) I add the files that I want to have committed with gid add, then I do a git commit, and then a

git checkout -- .

to overwrite the fluff in the working directory.

If you notice you’ve just committed nonsense (in this example I replaced the whole About This Database with nothing), you want to restore the file to what it was before. Here comes git checkout for helping again:

git checkout HEAD~1 -- odp/Resources/AboutDocument

HEAD~1 is a way to refer to a specific commit, i.e. ‘The commit on HEAD’ minus (~) one (1)

Conventions for naming files and directories:

dash dash

If you see before a filename, it’s a disambiguator saying ‘what follows are necessarily files’

dot

. is the current directory – so, everything.

Syntax for defining an entire directory with everything in it

odp/Resources/ would define that particular directory, along with anything inside it. Note the trailing slash!

Conventions for naming commits:

Naming conventions:

HEAD: Head is a pointer the commit that was pushed into the working directory last. Think of it as ‘commit corresponding to the current state of my working directory before I made any changes’.

<commit name>~1 The tilde here means ‘minus’, so this would be the previous commit.

Configuration options that are useful

In git Bash, I immediately increase the font size to something readable with our big screens, and then

git config --global core.autocrlf false

which avoids the annoying ‘lf/crlf’ comments, since I’m mostly working in a windows environment (sniff).
Do this before your initial commit! (see https://help.github.com/articles/dealing-with-line-endings/)

Setting up SSH keys with Git Bash

Since I’m lazy and don’t want to type in password and username every time I connect to git, here’s how to do it it cleanly via ssh.

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Press Enter for default in which to save key enter a passphrase (a complicated one, alright?)

Add the keys to the ssh-agent

 eval $(ssh-agent -s)
 ssh-add ~/.ssh/id_rsa

copy the PUBLIC key to the clipboard

clip > ~/.ssh/id_rsa.pub

add the ssh key to your github account Test the keys with

ssh -T git@github.com

Ignoring certain files or directories

There are some occasions when you don’t want certain files to be tracked by git. Typically, there are for instance binaries of your source code, or hidden files generated by your ide.

You can see which files are not being tracked with this command:

git ls-files --others --exclude-standard

or alternatively,

git add -A -n

There are unfortunately many different ways to define the excluded files. There is a central configuration file, whose location one can find with

git config --get core.excludesfile

Within your project, there are two files that control the exclusions:

.gitignore
.git/info/exclude

Confused? Here is an explanation from Junio Hamano (the maintainer of Git)

The .gitignore and .git/info/exclude are the two UIs to invoke the same mechanism. In-tree .gitignore are to be shared among project members (i.e. everybody working on the project should consider the paths that match the ignore pattern in there as cruft). On the other hand, .git/info/exclude is meant for personal ignore patterns (i.e. you, while working on the project, consider them as cruft).

This command combination is practical (thanks Akira Yamammoto)

git rm -r --cached .

(recursively remove all the cached files (i.e. tracked) from the current directory) then

git add .
git commit -m "fixed untracked files"

Notes/Domino best practices

Best practice when gitting an odp project: Never disconnect the nsf from the eclipse project.
Never change the nsf the odp is connected to.

Thanks to Adrian Osterwalder for the tips.