After some hard work we have finally gone beta with our service: RerouteMAIL. A service with which you can easily run a mailserver even if your ISP actively blocks port 25.
A while ago I posted about some wild idea's where you could go around the SMTP block of your ISP by running your mail through another (non blocked) mailserver and forward it to your mailserver which is running on a different port. Today, we go beta with a service which makes this possible for everyone!
The road from 'Proof of Concept' to 'Finished Product' was, as always, a bit longer than expected. First of all I switched from developing in PHP to Django (to expand my knowledge) and secondly, there are some pitfalls when you want to make this kind of service for more people than just yourself.
For the people who don't know what the fuzz is all about, a short rundown. There are ISP's in the world which believe that there are a lot of people who don't know what they are doing. So when those people run a mailserver chances are rather large that they don't configure their mailserver correctly, and they become an open-relay and thus, open for spammers to be used as spamming machine.
Although these ISP's are in their right to do so, we believe the clients (and especially the business clients) should have the chance to be able to run a mailserver if they completely understand the risks.
By default (and according to everyone and everything on the internet) a mailserver listens on port 25 for incoming mail messages. Because this is the default, ISP's block this port (incoming and/or outgoing).
Fortunately we have thought up a way around the ISP's who are actively blocking port 25.
As a lot of people (mostly unix administrators) know, you can run a mailserver on a different port than port 25, say port 26. However, the email world does not know this. So when a mailserver (for example the google mailserver) wants to mail to you, it tries to connect to port 25. And so you have to find a way to let the other mailserver think you are on port 26. But this can't be done easily and without your ISP.
It can be done via our new service: RerouteMAIL. The trick is that we listen on port 25. And then relay the incoming message to your mailserver running on port 26!
And the best part is, we have a fee free version which everyone can use, a hassle free registration, and no questions asked.
If you are interested or curious please have a look at: www.reroutemail.com or leave a comment on this blog!
It really is everywhere! Until now, in every company I have worked for, I have seen it. Projects fail to hit their deadline. And total anarchy emerges from the ashes of those failures. Save yourself (or your company) while you can and learn from the top 10 list of reasons I came across in the passed 8 years.
I have been developing (web)applications for some time now. And although some people might consider the 8 years as being "new", I consider it long enough to be able to point out some important reasons to why projects fail.
Although it was really really hard, I tried to make up a top 10 list of things that can (and have gone!) wrong. And because if I have experienced these problems. You can bet others (you?) have and will too! So, here goes. My top 10 reasons:
10. Not all things have been taken into account
It sounds too obvious to be true. But it is. When people start a project they mostly don't take time to step back for a moment and consider the implications of the project. I haven't been in a meeting which went like this: "Hey! Is this really as easy as it sounds? Or are there hidden problems lying low and ready to jump at us when we least expect it!?". Most of the time people are too excited about the project and feel like they can take on the world! And they realize halfway through the project (if you're lucky!) that they didn't realize feature X would take as much time as it did! And so, their deadline creeps up on them and BAM... The deadline can not be met.
9. It's a rush job!
How many times haven't I heard this one: "I know it's a rush job! And we are aware of it!". Only to hear a few weeks later: "But how is it possible that this bug exists?! We just CAN'T have these kind of bugs!". And most of the times these bugs only surface after it has gone live for a few days.
8. There are no milestones
No proper milestones have been set for a project. And this is not good. Not for you, nor for the customer! If you don't have milestones, you have no way of measuring your progress. And even when you think you are on the right course, you can never know for sure. More so, you cant update your client (or customer) on the progress. Because you just don't know!
6. The customer lacks technical insight/knowledge
It can be a real problem when the customer (or your manager) is not as technically endowed (on this subject) as you are. But realize it is as much of a problem for them as it is for you! They wonder why you take so long to create feature X which seemed soooo easy to them. The biggest problem is not their perception of complexity, because this you can explain. The biggest problem is that they will not be able to explain their wishes to you in detail enough so you can make a correct estimate on the project length.
5. Ad-Hoc projects / Prolonged FireFighting Mode
Your morning at work starts and your manager runs into the room yelling and screaming: "WOEAAAH!!!!!! WE NEEEEEEEEED BUG 1337 FIXED RIGHT NOW!!!!". The sky has fallen! All customers have ran off and the cash has stopped pouring into the bank account of the company! All because of this bug! Or... so your manager led you to believe! You rush yourself to your computer and work long and hard to fix the bug as soon as possible! Finally! the bug is fixed! And you can continue your daily work on bug 1884. But then, after lunch, it starts again: "AHAHAAAAAAAA!!!!! THERE IS A PROBLEM WITH X AND Y! THIS CAN'T BE HAPPENING!" and of course you need to fix it right away! This scenario continues some time.. Until at the end of the day, the things you had to work on that day didn't finish. A day passes by and you fix some "MUST BE FIXED NOW" bugs until one moment: "WOEAAAAH! WE NEED BUG 1884 FIXED!! AND IT SHOULD HAVE BEEN FIXED 2 DAYS AGO!". And the circle is complete... Because you were fixing bugs which seemed critical, you are not able to work on your daily routines. Which in turn, turn out to be "need to be fixed now"-bugs eventually.
The biggest problem with this is that you will always keep a feeling of behing behind all the time. And escaping from this vicious circle can be very hard. Although you can't prevent this problem directly, you can discus the matter before fixing the "need to be fixed now"-bug. Ask your manager "Is this 1337 bug really more important than bug 1884? And if so, are you okay with delaying 1884?". This will probably make your manager think the request over and it will allow you to fall back on the conversation when 1884 became a real "need to be fixed now"-bug
4. Communication problems (or lack of communication) "
It happens quite a lot. A salesperson has promised feature X to customer Y and has told the Project Manager to make it happen. It slips the mind of the PM and the salesperson didn't remind the PM about the feature. Until the time approaches that it should be delivered to the customer and the salesperson asks the PM about this. "Damn!". The PM wraps up some developers and a technical lead and you'll hear something like: "We have promised the customer a month ago we have feature X in the release for the end of this week so build it and build it right!".
Everybody should know this is a recipe for disaster. You just can't build a feature in a week (or less)! No arguments. You just can't!
3. Total lack of specifications
You think you have done everything right. You talked to the customer, written things down, had the customer sign the document on what to deliver. But you don't write the specs. You didn't work out the details of the features and think you can manage it along the way.
Just Face it! In the larger projects (even if it is still a one man job) it is almost impossible to do things right the first time. And unless you have calculated a lot of refactor time (and I really mean a lot!) into your project, your deadline's will slip. And you will have an unhappy customer. Just take your requirements and work them out. Just make small mock-up of what you are going to create. Write down some flows. And be sure to show them to your customer and have them comment on them.
You don't have to create full detailed Interaction Designs. Work out all the Use Case Scenario's and write pseudo code for every function. Just make sure the outlines are there and have them approved or reviewed.
2. A Sales person determines the deadlines without consulting programmers
This I hate the most!
Operations come's to you and says: "Yeah we have to create X, and it has to be completed before end of the week!"
"Que?"
Damn, they have done it again! They promised stuff to the customer, and we don't even know if it is technically possible to do so! And even if it is possible, we don't know if we can have it production ready in that time...They haven't even discussed it with the PM or a TL.
1. There are no, or incomplete, requirements
This one is also very classic, and very easy to make. "We need a program that can do this and do that, <explanation of what and how etc>.. can you make it?". Sure you can! And there you go happily programming and creating what the customer asked. And after some hard weeks of working and finally delivering the program, you receive the following reply: "Ah, yes.. But... This is not how we wanted it. It should work like this and that!". So you reply: "But.. Why didn't you say so?". And the customer replies: "Well, er, it is obvious that it should work that way!". And there you go.. Throwing away a big part of the program, starting over again..
And the biggest problem is, there is really no one to blame! It's just that people won't exactly, with 100% certainty, know what the other person really means or thinks. So, even when you have requirements, it is quite hard to say "Well we decided this and that and I made it that way. So, its your fault!" because with that attitude you will only get your customer mad and create an atmosphere in which you can't work.
The solution? Make sure you agree on what is required of the project, and make sure you write them down. Have your customer specify what the program has to do to make it acceptable for him. And finally make sure you live the project as much as your customer does!
Well there they are! My list of reasons why a development project can and will fail. They are based on my personal experience in the passed years.
Have you experienced these reasons as well? Or have experienced others? Leave a comment!
You might recognise the problem. You have a close friend group and you do lots of things together. So when you go out, or go to the movies you decide that you'll pay something, and the others pay other stuff and you dont realy worry about the money. They are your friends, and you trust everyone, so, no biggy.
Now although this is true, the problem lies where you can't keep track of what you owe who, or who owes you what. And although you know it's okay, it just doesn't feel right. Especially when it concerns larger amounts. For example, you go on vacation as a group. Decide that everyone will pay amount X as a money stash for group activities. But you know it is a matter of time before the plan becomes a bust. Eventually someone forgot his card and could not make a withdrawl. The account was empty sooner than expected. Etc. etc. In the end person X has deposited €400 person Y €600 and peson z only €200 because of whatever reason.
Now I have, on multiple occasions, decided to make a nice little Excel sheet which can handle these kind of problems. You enter the money you have spent (its a friend case so you trust everyone supplies accurate numbers) of each of the members, and it calculates neatly what the average amount spent is, and you can see who owes what to whom. Further, if person Z afterwards gave person X some money for something else, it is handy that these numbers can be specified too. So it will always reflect who has already settled his debt and who hasn't. The biggest problem is you have to manually keep track of changes. If someone pays somebody (partially or full), and didnt notify me, I can't keep track of the changes. And so it is a matter of time before it is not accurate anymore so you can't rely on it.
Because I (we) have these problems. I'd think I'm not the only one! So I think about making some simple application which can handle these kinds of troubles. Attach a mail service to it so people can easyly be reminded of his debts or request for information. Combine this with some type of grouping of friends and you have a centralised application to handle all current and future money oweing problems without worrying about who paid who (or didnt).
You have an ISP which is just fabulous, great speeds, acceptable pricing and no download limits. Great! Now you can have your own linux/windows server right at home. Run a webserver, claim some domains, host a few other things (mysql, python, ruby, django, etc) and tinker around with everything you couldnt do at a hosting provider. But when you try to configure your mailserer the fun ends. Mail just doesn't get deliverd... Booo!!!
At first you think you configured your mailserver wrong. Maybe you didnt set up your MX record corectly for your domain or maybe, just maybe, your postfix config is wrong (what could go wrong there? :P). After some tinkering and frustrated tests you come to the annoying conclusion that it is your ISP who is nagging you. It appears they block the SMTP port (port 25) in their firewall, so nothing gets through to your server. Damn it!
So your first thought is to go around it. You are a great tech-buff and you think you can outsmart your ISP blocking a port in its firewall and find a workaround, so you think to yourself "why not run it on port 26 (or something else) instead??". A fine idea indeed. But there is one problem. SMTP servers don't know your mailserver is running on a different port. And it has to know!
Let me illustrate this a bit more.
When someone (user@gmail.com) typing a message and wishes to send you an email (user@somedomain.com), creates the mail, and clicks on send, the google mailserver first contacts the DNS server to request the MX record for the domain (somedomain.com). It receives a response in the form of an IP address.
Then the mailserver will connect to this IP addres on port 25. Why port 25? Because thats the way all SMTP server work. It does not know any better than to connect to port 25. Its the way SMTP works.
So, no dice.. Just using a different port is not enough (and No: tricking the world into thinking port 26 is the new port 25 aint gonna work).
After some thinking the solution came to me. If we can't trick the world into sending to port 25 and we can't go around the port block at the ISP. Why don't we run it before the ISP or at another ISP?
"Uh, but, aint that the most obvious solution?"
Yes ofcourse it is. Running your mailserver on a different location where port 25 is not blocked is the perfect solution. But it is not appropriate in this case indeed. You want to run you own mailserver at home so you have full control over it. Not at a colocated place, hosted by a hosting provider and what not.
But think one step further:
"If someone would be willing to run a mailserver outside of your ISP's influence, and forward all incoming mail which is supposed to be for your domain and forwards this to your machine which has actually running a SMTP server on a port other than port 25 and delivers the mail on that port... Wouldn't that solve the problem?"
And there you have it. The solution to the problem. This way you can run a mailserver yourself even when your ISP blocks port 25!
Now I've been searching around for someone who offers this kind of service, but was not succesful. So I thought to myself, why not create such a service yourself? And so from the need of a solution, there came a solution and maybe a solution which can help other people aswell!
Currently I have this problem solved in a proof-of-concept scenario where a friend of mine is running a SMTP server (on port 25) and he forwards all emails directly to my SMTP server running on port 25. Right now I'm looking into ways of providing this solution to other people.
Update: Recently we (Marc-David and I) have launched our service at http://www.reroutemail.com/
I recently started working on a new freelance project. Now there's nothing special about that because thats what I do for a living right now... But reading around on the internet and in books made me realize again that blogging about it is a good way to get things 'out there' and a way to clear my own mind.
My new project involves the creation of a pretty straight forward web application. Users logging in, an amount of relations between different objects. Some listings, some variables, some conditions, the works.
The fun part about this project is that, so it turns out, it is the stuff that realy makes me happy doing. Its a project in which I can do everything from start to finish. The basic idea is provided, a basic layout is provided and most of the data which should be available stored in the database is provided. So it basicly comes down to creating a solid DB design, creating a technical design, PHP coding, HTML layout, bug testing, bug fixing and deployment.
There are no real constraints on this project in terms of baggage from previous made stuff. No buggy feature which was previously implemented and results in quircky behaviour. No strange DB Designs which actually don't make sense at all.
Most projects I have done in the past actually have this kind of baggage. I am constraint in how to set up the project and only have the freedom of when I do the work, not in how I do the work.
But with this project, I have total freedom, and it makes me happy!
Right from the start I figured I could again use my freshly created framework which, supposedly, 'failed' according to a previous post. Now I couldn't be more wrong when I wrote it! It turns out that although I hadn't much use for some of the features in the past, the things I had created came in real, REAL handy! I had a basic site up and running in a few minutes and in a day orso I was back into the feeling of how it was to code and make real progress real fast. The need for improving the framework returned. And, more pleasingly, my motivation skyrocketed and the work wasn't work anymore, it became fun.
For me fun is one of the things needed to stimulate progress and creativity. When it is fun to work on a project you are constantly thinking about how to create new stuff and improve stuff. In this project I have fixed more bugs and created more effective functionality in the framwork than when I was actively working on the framework itself. And fixing stuff gives you pleasure/satisfaction, and pleasure drives you to create/improve new things, which in their turn gives you more pleasure/satisfaction. It's a win-win situation. And it is one I realy needed at the moment!
Expect more posts later on the project (and I try to keep my word this time :))
After a few weeks I decided to start reading "Getting Real" from 37signals. I bought the book when I read about it on a blog from Scott Berkun (Why Requirements Stink) .
I started in the morning and read it from cover to cover in a few hours.
The book does a fabulous job in creating a 'think small and easy' mind-set. And it does so without going in too much detail. Every chapter is relatively small and gets to the point quickly. After every chapter I got the feeling that I wanted to put the theory into practice and start right away. But moreover it made me wanna read the next chapter and find out more (which ultimately resulted into me reading it from cover to cover in one read).
As stated in the introduction, a lot of the principles and methods are not thought up by 37signals themselves. They borowed a few here and there and added their own sauce to create a 'methodology' which is very suitable for themselves. Although they state things as being a fact, they never force things on you. They just point out the way they do it and why it helped them to achiever their goals.
The book made me realise again how 'easy' it is to start a web application and do it right but without all the fuss of elaborate planning, specifications, etc, etc.
Another side-effect of reading the book is that I wanted to blog about it again, and it resulted in me writing 3 blog posts today. I just wanted to get it all out of my system, and get it out of there right away. True, there may be typo's and grammatical errors in some of the posts, but it is the web! When someone points them out they are realy easy to fix. Thats the way the Internet works! 
In one of my earlier blogs I posted about the framework I wanted to create. At the time I was really anxious in creating it and making it as cool as it was in my mind. Unfortunately, all didn't go as planned and although I have really created some nice stuff, it was not as I had expected.
So what went wrong?
First of all, I stoped working in an environment where the framework was needed for. I stoped working at my former employer and started my own business. At this company there were all sorts of problems which could be fixed by creating a framework of tools and guidelines. The framework had a purpose and the features it should have where crystal clear.
But when I stoped working there, this problem went away... So what I was eager to create became a "solution in search of a problem". And because the necessity for it went away, there was no drive to create all the aspects of it. With that, usefullness and motivation went away.
For a long period of time I did not really see this. I happily started working on it, but was not getting the results I was hoping for. Over time, the time spent on the project deminished and time was spent on other projects instead.
So the project failed?
Well, not really. I did create a solid base for all my other web development projects. The simple websites I created use the framework and it really helped me create them nicely and easily. Right now I use the framework as a "means to an end" instead of an "end". When the framework does not support something I want, I create it. Not the other way around. And although I should have learned this lesson before, from sources like Scott Berkun's: The art of Project Management and numerous other sources (books and online). I think the only way someone really learns the meaning of a statement like "A solution in search of a problem" is to experience it first hand. Preferably in a way he fail's (failure is the best learner).
What now?
Well, I learned quite a bit from the experience, and right now I am aiming on some new projects and products I would like to create. A lot of idea's sprung from reading "Getting Real" by 37signals. I will be posting about the book, and the idea's I'd like to work out, pretty soon.