Mostly Harmless

I’ve spent the last day trying to get support for a Tracks install on Servage that went from running a little bit slow to not running at all, here is what I found out. It also provides a little insight into how not to deal with customers problems with third party scripts.

Wasting time managing time

I’m very interested in Rails but we don’t offer it yet at work because it isn’t an amazing fit with a CPanel based service as it is a lot easier to install and maintain with shell access. So apart from having played about with it on my local machine I hadn’t actually been into how it works on a server in any great detail.

I’ve been using Tracks (a time management application) to make me do all those little things that tend to get forgotten for the best part of a year now and providing you log into it everyday ::) it does actually help. The install on Servage never ran fast, but it worked and that was enough.

Over the last month the install has become completely unusable. I submitted a support ticket to servage indicating the virtually constant failure and requesting assistance.

Answer me these questions three…

The first reply came one hour forty five minutes later..

I have made the changes to make your rails app run.
I’d suggest you to use fcgi, with is about x10 faster then normal cgi.

On the surface there is nothing wrong with that response (in fact it contains the root of the problem) but it does highlight a couple of important things about giving customers replies they can work from

1. When you make changes to a customers install of a script, let them know what the changes are
2. If you’re suggesting making changes, explain the changes and how to action them, however briefly

The app now indeed did run again, however it was much slower than it had been when working in the past and occasionally completely timed out. I reported this and was asked to upload a .fcgi handler. This confused me because an fcgi handler is already part of the app and clearly visible in the webroot of the application - I mean, surely they’d looked at the app when troubleshooting it? (As a support tech myself I find looking at something a prerequisite to fixing it).

Nevertheless I provided them with the path to the file. 40 mins later…

“May I request you to please upload the tracks application under /(the whole tracks app as /tracks).

The current one seems to be having a problem with incorrect handlers. “

Again pretty confusing given that I’d not modified the install of Tracks in any way since installing it and the readme files for it were all still in place. Also, what are the details of the problems with the handlers?

I’d gone to bed by now but a couple of hours later the ticket was updated with

“Please do check once again and see if the app wworks fine from your end. “

So I presumed they’d found whatever the issue was (still not giving me any details mind) and attempted to login to the app. No change

At this point I decided to upgrade to the latest version of Tracks and have a read of the docs before replying again. I figured that possibly the .htaccess wasn’t directing to the correct handler (as per Servage’s first reply) so asked them to check it. The response I got was what I call a typical fob off

“Web programming providers a large amount of flexibility but can be hard learn and small mistakes can make a script malfunction.

Unfortunately it is beyond to debug scripting issues that is not related to server problems or server misconfiguration. Instead we recommend either to contact the person or firm who created the script.

Alternatively you could seek help in one of the many programming communities on the Internet as it is a create way to get help and improve your scripting skills.

Link:
http://forums.devshed.com/

I hope you find a solution to the problem you are facing“

Now sometimes it is wholly legitimate to issue a “fob off” - normally when you’re damn sure the problem isn’t under your control and ideally you’ve provided proof of this and yet the client still doesn’t get it. The worst reasons for providing a fob off are “we don’t know what the hell is going on” and “the person who knows about that isn’t on shift right now and we don’t want to admit we don’t know”.

It annoyed the hell out of me because I hadn’t been issued with any proper details on why the app didn’t work. I replied asking for error messages and what action they had taken earlier to allow the app to work again as they’d clearly made some improvement to it.

I was told the ticket would be forwarded back to the tech who had originally looked at it.

A day later I decided to roll up my sleeves and try and understand what was really going on. As far as Servage goes there seem to be two methods of running a Ruby / Rails app, cgi or fastcgi (remember that first response? you can get where I’m going with this can’t you?)

The long and the short of the problem is that the Rails app needs to be instructed via its .htaccess to use the correct handler, which in the case of Servage appears to be

AddHandler fcgid-script .fcgi

The other fcgi handler AddHandler fastcgi-script .fcgi should be commented out.

Secondly you need to edit the re-write rules further down the file so that they are using dispatch.fcgi rather than dispatch.cgi - simply replace the file extension as necessary.

RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

I’ve replied to Servage to let them know this and to be fair they provided me with the right answer in their very first reply - however if I were “Joe PA” setting up a time management app for his boss that wouldn’t even have given me a hint of what was wrong.

What is totally puzzling is that Servage seemed to know the answer, but couldn’t link that up with checking the rules in the .htaccess for the application. Despite someone at Servage at least suspecting the cause of the issue, two other techs tried to tell me I simply needed to learn to code properly - when if they’d even taken a cursory glance they would have known it’s a reasonably well known open source application.

Because I’m hazy on the way you implement Ruby with Apache I don’t know whether it’s weird or not that Servage provide a Ruby cgi implementation that is too slow to actually be of any use.

Somewhat after the fact in my book I receive another response with some error code!

public/dispatch.fcgi
public/dispatch.fcgi:19: uninitialized constant RailsFCGIHandler (NameError)

Running dispatch.cgi never stops.

Please contact the support of the software. It is too complex for us to debug.

Which completely ignores my previous reply. Granted they’ve finally produced an error message (whoopee) but also now the app doesn’t work at all - all the files have been changed to webserver ownership and the .cgi line has been added back into the .htaccess. I reply asking if they’ve changed anything.

I leave it a while and then change back the ownership of the files and comment out the RewriteRule ^(.*)$ dispatch.cgi [QSA,L] line that has been added - and things start to work again. I then get a final reply:

“I have checked the application settings and they seems correct. There is however a problem with your app, it runs in a infinite loop”

I open up the .htaccess file and the line I commented out is gone entirely. The app at this point works but with occasional 500 server errors.

The issue I have with the last reply is that from what I know of programming - if an app gets stuck in an infinite loop whilst doing something, then it always gets stuck in an infinite loop while executing that function. In this case the error is sporadic - which makes a coding error much less likely as the app is not really that complex. Generally if an app enters an infinite loop while displaying its login box, it isn’t only going to do it half the time.

And then the light strikes the right place - the database server for the blog goes offline as I write this post, is it the same server as the tracks install? Damn right it is. Switch the database to another server and bang - fully working tracks installation. It appears that any delay or failure in the connection to the database was making the app fall flat on it’s face.

This blog occasionally suffers database connection errors so I’ll be moving it’s database as well.

Conclusions

The first thing I’ve learnt over the last couple of days is that providing support to a clustered environment can clearly throw up complex issues with hosted apps - I’ll bear that in mind if / when we move to clustered setups at work. I pity the poor customer service guy who has to work off a troubleshooting flowchart system rather than his own initiative and knowledge who comes across an issue like this…

Secondly it reiterated some things I already knew

1. If you’re a web host, your support techs have to be able to know how to configure something from a client perspective and be clear on what the client can and can’t see. Providing error log output is important, making sure the users app is configured correctly is too. The first thing we do at work when someone says “my app doesn’t work” is to check their config files (like .htaccess) because nine times out of ten that’s where the problem is (as was the case here) when you know your environment well you can quickly fix the problem and impress the user. Yes you are in effect supporting a third party script (something against most Web Host terms and conditions) but it costs you less than a discussion that might last days and the client stays with you. Tell the client what you did and next time they’ll do it themselves (hopefully).
2. Don’t be afraid to roll up your sleeves and get into the documentation and try different options if support seem clueless.
3. I really ought to have something where I’ve got full root access to host my sites on.

To be fair to Servage they had the answer in the first reply and I freely admit I could have configured the app better in the first place.  They also remained polite throughout. But I also got a few replies which come over as unsubstantiated fob offs where a longer investigation of the problem would have been appropriate (because lets face it I wasn’t going away till it was fixed).

Still, I know a fair bit more now about how Rails apps execute now…

Rowan :: Mar.29.2007 :: Web Apps, Web Hosting :: 3 Comments »

Trackback this post | Feed on Comments to this post