Sunday 15 July 2007

Look Ma: No Engine!

I took my engine out for the first time yesterday. It's not that hard actually but don't tell Dee because she's under the impression that my voice is a little deeper and my underwear a little less roomy today.

Vol 3 of Bug Me Videos shows how to do it but Rick fails to mention what to do when you're up to your elbow trying to undo one of the top nuts that hold the engine into the car and your other arm gets attacked by the world's most determined horse-fly. The little bastard had me dancing all around the garden trying to escape it before I eventually managed to clap my gloved hands on it - a shame really because in my new found masculinity I might have mounted it's head on a plaque in the hall had it been in one piece.

Another small problem was that the same top nut was a little reluctant to come all the way off the bolt. It came far enough that the head of the bolt would release from it's locked position and turn with the spanner. Most irritating, but with my right hand reaching up through the wheel arch to push the bolt back and my left hand on the spanner I eventually got it loose.

The final problem of the day was the when I came to lower the engine onto the dolly that I got from VW Heritage and discovered that the wheels of my 3-tonne jack were too wide to go between the wheels of the dolly. It hadn't occurred to me to check. Luckily the rear valance on my bug is bolted into place as opposed to being welded so I didn't need to raise the car up as much as Rick does in the video before pulling the engine back and was able to use my 2-tonne jack to lower the engine the necessary few inches onto the dolly.



At this point I was able to remove the louvered panel attached to my firewall and unclip the wiring that runs behind it. Although that was mission accomplished (the whole reason for doing it was that I needed to get to those wires) I obviously had a good look around and while I was at it I found the bits shown below inside the clutch/flywheel housing on a little ledge next to the gear from the starter motor:



Obviously I was shocked because no way should there be any bits of loose metal flying about in there and judging by the little marks on the inside of the casing, they had indeed been flying around at some point. I posted about it on VZi and Moby5153 identified the clips as being transit clips for the clutch pressure plate that should have be removed after clutch installation. I already had evidence that the garage who fitted the engine didn't really know what they were doing and this merely compounds it.

As for how the hell a spade connector got in there...

Friday 13 July 2007

Rust Sandwiches

Although I've made a number of posts relating to my research and planning, I haven't said much about the actual work in progress on my Beetle. Alas things are not going too well. Bad health, bad weather, and a need for out of stock parts have all played their part in delaying things however the largest factor has been that I keep discovering even more bits that need replacing.



I knew when I bought her that the left hand heater channel was in poor condition. I was quite pleased with myself at the time for spotting it because it wasn't glaringly obvious and in a matter of weeks, I'd gone from knowing almost nothing about cars to knowing enough about Beetles to spot it. However I failed to spot that during a previous repair, the heater channel had been welded to the floor pan. It's really obvious when you see it but I just didn't notice and the fact is that it's the same on the right hand side meant that both sides looked the same and didn't alert me to it.

That in itself is not a huge problem but in the process of removing the left hand heater channel and floor pan half I've also come to realise that the corner of the front-head base-plate is rotted away, along with the end of the front chassis support, where they bolt to the heater channel. At the back, the outer part of the rear cross member is rotten and there were so many bad areas lurking beneath thick layers of underseal in the rear wheel arch (covered by glued on carpet on the inside) that it probably makes more sense to replace the rear quarter rather than repair it.

Now while all of that is possible, it would be a heck of a lot easier with the body off. However, while I've got replacement floor pans for both sides and a new heater channel for the left hand side, the heater channel for the right hand side is out of stock. Remember that the one on my car is welded to the floorpan and you'll relised that I can't get the body off and make the car roadworthy again before I get my hands on a new right hand heater channel.

Vol 7 of the Bug Me Videos shows them removing a heater channel in one piece, and while that makes a lot of sense for demo purposes (and let's face it the new one is going to have to go in in one piece) it's not necessarily the most practical way to do it if there are 'unknowns'. I've already said that mine was welded to the floorpan but in the video they say that there are no welds along the top of the channel where it meets the front wheel arch on the inside of the car; just spotwelds on the outside. There clearly was a weld on the inside of mine and things became even more interesting when I started to cut it out. It turned out that there were four, yes four, layers of metal.

On top of the original, rusted to bits, heater channel was a patch. Presumably this had also started to rust at the bottom at some point because there was another patch on top of that, and another patch on top of that.

Now that I've got it out of there I can see that the bottom part of the front wheel arch also has a patch on the outside, plus a patch on the inside, and the original rusted piece of metal is sandwiched between the two. Clearly that's going to need replacing too.

Apparently there's a saying in VW circles about having replaced the bottom six inches of the car. I seem to be discovering what that means.

Tuesday 3 July 2007

Opening Ports With PHP

In my previous post I described how I'd set up iptables. The important thing to note about that is that I set my default INPUT policy to DROP anything that isn't specifically accepted by a rule. This means that I can easily open and close ports using the iptables command to add or delete rules. For example, the following can be used to open the port for webmin:

/sbin/iptables -A INPUT -p tcp --dport 10000 -m state --state NEW -j ACCEPT


I can then close it again by deleting the rule with:

/sbin/iptables -D INPUT -p tcp --dport 10000 -m state --state NEW -j ACCEPT


Note that:

1. Webmin is always running, and listening, even when the port is closed, so I don't have to issue commands to start and stop it. I just need to open and close the port.

2. I don't have to leave the rule in place for the entire session as the commands above allow and disallow NEW sessions. Other rules in my iptables setup allow established sessions to continue. (Although in fact Webmin tends to stop and start the session as you use different elements of it so it's as well to leave the port open until you are done.)

3. The iptables command can only be used by root.


Now I'd like to be able to do the same thing with SSH but of course there's a catch: if I close port 22 then I can't SSH in to issue the command to open it. Don't get caught out by that one!

However, as mentioned in a previous post, my intention was to use a secret backdoor by getting one of my .php web pages to watch out for a special input and use exec() to run the iptables command. However there is a catch here also: the PHP script runs as apache but the iptables command can only be run by root.


Now there are a couple of ways around this and what I've chosen to do is to use the sudo command. This allows a user to run a command as another user however they have to be given permission in the /etc/sudoers file. I did this by adding the following line to the file (using visudo to edit it - as it tells you that you must in the file itself).

apache ALL = (root) NOPASSWD: /sbin/iptables


This gives apache the ability to run the iptables command without the need for a passwork. Shock! Horror! Isn't that a security problem?

Well, not really, the addition allows apache to run iptables, and that's it, nothing else. It's also important to realise that on my server, you cannot log in as apache, and in fact I am the only user allowed to log into my server at all. I am also the only person who can upload .php files to my server so I am the only person who could install a .php script that uses exec to run iptables as apache. Now even if somebody else did figure out a way to get around that, 'all' they would have achieved is the ability to open and close ports. They'd still need to crack other passwords before they could do anything useful/nasty. Furthermore, the instructions being issued to iptables are reported in my Logwatch report so I'd be made aware of it.

Of course if you don't have the luxury of being the only person who needs access to your server then you may be better to consider another approach. For example you could use your php script to create a file or change a setting in a database to act as a flag for a cron job. You would then create a cron job, which you set to run every couple of minutes, to check the flag and run the iptables command when the flag is set. The downside of course is that after setting the flag you have to wait for the cron job to run before you can get in. The upside is that apache no longer needs the ability to run the iptables command. Apache just sets the flag and the cron job (which you run as root) issues the iptables command.

Incidentally, there's a heap of info about the sudoers file available by typing 'man sudoers' at the Linux command line with loads of examples down at the bottom of the man page. Check it out and you will see that you have a heck of a lot of control over what you do and do not allow sudo to be used for. Given the various other restrictions on my server, I'm happy to allow apache to use sudo to run iptables and can therefore use the following PHP to open and close a port for SSH when I tell it to:


// create an iptables rule to allow access on port 22
exec('/usr/bin/sudo /sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT')

// delete the iptables rule to allow access on port 22
exec('/usr/bin/sudo /sbin/iptables -D INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT')


Note that my iptables setup (as I described a my previous post) is such that port 22 is open when my server boots, and I leave it that way.

Under normal circumstances, after rebooting the server, I would tell my PHP script to close port 22 and would then leave it closed when I'm not using it. However, I leave it so that the default after a reboot is to have it open. This means that if something should go wrong such that I can no longer access my PHP script to open the port (which I would need to do to change the PHP script), I can get it open again by requesting a reboot.

Watch out for this:



Perhaps the biggest headache that I had in setting this up was that initially I couldn't 'sudo' via exec(). I spent hours looking for the problem and it was driving me crazy because everything I found on the subject indicated that what I was doing was absolutely spot on. The breakthough came when it was suggested that I try this:

echo exec('/usr/bin/sudo /usr/bin/whoami 2>&1');


The result should be 'root' but what I got, courtesy of the bit on the end, was this: 'sudo: sorry, you must have a tty to run sudo'

A search on that told me that the issue related to a setting in /etc/sudoers which is used in Fedora Core 6 namely:

Defaults requiretty


I didn't get entirely to the bottom of why it's there. Something to do with there being circumstances when the password you are entering for sudo can be visible on the screen. Apparently it's a new thing in FC6 and all the advice I saw suggested that the solution was to comment it out (which leaves me wondering if that setting will still be present in FC7). Either way, I'm using sudo in such a way that I'm not typing a password so I don't see that it's an issue.

Monday 2 July 2007

iptables

iptables is something that I've steered clear of until recently. Partly because it looks blooming complicated (and why bother when scripts such as system-config-securitylevel will set up a firewall without getting involved in the nitty-gritty), but also for fear of locking myself out of the server (which is 100+ miles away at a server farm and doesn't have a screen and keyboard attached). When I became interested in port-knocking however, it seemed that the time had come to do a little research.

The first bit of good news came when I discovered that you can 'mess' with iptables without making the changes permanent. When the server boots and iptables sets up the firewall, it loads rules from /etc/sysconfig/iptables however you can then change those rules from the linux command line using the iptables command. In fact you can create a script to delete the current rules and load a new set. The crucial thing here is that you do not edit /etc/sysconfig/iptables directly. Firstly because it's 'system generated' and secondly because if things go wrong, you can reinstate the original set of rules by rebooting the server. While I can't get physical access to my server, there's a mechanism by which I can request a reboot. Of course when you have a set of rules that work you will want the server to use them when it boots and you do this by issuing the following command causing the rules that are currently in memory to be written to the /etc/sysconfig/iptables file:

service iptables save


Safe in the knowledge that we can dabble, it's time to look at creating the script that will install our rules. The first three lines look like this:


#!/bin/sh
/sbin/iptables --flush
/sbin/iptables --delete-chain


Obviously the first line starts our shell. The next two lines clear out any existing iptables rules and user defined chains (of rules) that are currently in memory.

iptables works by looking at packets that are wanting to cross the firewall. These could be packets coming into the computer from the network, going from the computer out to the network, or being forwarded (if the computer is also being used as a router). Because of this it is normal for a set of iptables rules to start with a set of default policies such as:


/sbin/iptables -P INPUT DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P OUTPUT ACCEPT


I've seen examples sets that work on the basis of accepting anything that doesn't get blocked by a later rule, and I've seen others that block everything unless it's accepted by a later rule. I've gone for a mixture. Nothing comes in unless a later rule allows it. My server is not being used as a router so all packets for forwarding are dropped. All outgoing packets are allowed.

It might seem odd that you would want to do anything other than allow all outgoing packets however in a scenario where the computer is allowing a number of machines on a LAN to connect to a WAN, you may want to implement restrictions.

The next thing is to set up the loopback interface:


/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT


This allows local traffic such that daemons running on our computer can send packets to other daemons running on our computer via the firewall. Stricly speaking, we only need the first of the above lines in our script because our default policy for output already allows outgoing packets from our daemons.

With the default policies set up and our daemons able to talk to each other, our computer can see out, but nothing can see in. If this were a home computer being used to access the web, we could stop right here, however I'm creating rules for a web server and there's no point having a server if nothing can access it.

Before we create a few rules to open things up, it's important to understand that we can split and packets reaching the interface into one of two groups: those initiating a connection, and those that are part of an established connection. This simplifies things because we can allow packets for established connections to pass through unhindered because we already did all our checks before we allowed the initial connection.


/sbin/iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT


The first line says that all new TCP connections must start with a SYN packet i.e. that they must start properly. The second line says that packets for established or related connections are allowed through. If we stop there then we will still be blocking all input because we haven't allowed anything to establish a new connection yet.


/sbin/iptables -A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 25 -m state --state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT


The three lines above allow new connections by accepting packets coming in on ports 22, 25, and 80 i.e. SSH, SMTP and HTTP. Note that this does not mean that anybody can connect via SSH. The still have to be on the list of allowed users and they have to know a password. The above simply means that the port it 'open' such that it is possible to connect.

We're almost done now although there are lots of other things that could be done with our rules and that you will see in other suggested iptables setup scripts. Sometimes you may need other ports open and sometimes you may want to do things like blocking traffic from IP addresses that are being a nuisance. You can also do things like locking out an IP (for a period of time) that has made more than a given number of connection attempts in a given amount of time. In a later post I'll describe describe a technique that I've implemented to greatly enhance the security of my server using iptables rules but in the meantime I want to make just one final addition:


/sbin/iptables -A INPUT -p ICMP --icmp-type 8 -j ACCEPT


This accepts incomming type 8 ICMP messages and allows people to ping the server. If this were a desktop computer from which I wanted to issue the traceroute command then I should also allow type 11 ICMP messages. However it isn't, so I won't, and our final script looks like this:


#!/bin/sh

# Flush old rules, old custom chains
/sbin/iptables --flush
/sbin/iptables --delete-chain

# Set default policies for all three default chains
/sbin/iptables -P INPUT DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P OUTPUT ACCEPT

# Enable free use of loopback
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT

# All TCP sessions should begin with SYN
/sbin/iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

# Accept inbound TCP packets
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 25 -m state --state NEW -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT

# Accept inbound ICMP messages
/sbin/iptables -A INPUT -p ICMP --icmp-type 8 -j ACCEPT


Of course the final steps are to save our script, make it executable, and run it as root. We then need to check that everything works as we expect. Most especially that we can still initiate new SSH connections to our server. When we are happy we use the 'system iptables save' command to write our configuration to /etc/config/iptables and we're done.

Sunday 1 July 2007

Thinking About Port Knocking

In a previous post I described how I'd tweaked my server's security with respect to SSH and Webmin however I really wanted to make it more difficult for a potential abuser to get anywhere near them. I also wanted, if possible, to rid my Logwatch reports of the hundreds (sometimes thousands) of lines reporting failed attempts to log in via SSH.

Using a non-standard port for these services is a commonly suggested 'solution' but while this may throw most of the current crop of bots off the scent, any half-serious hacker can easily do a port scan first. You can also be fairly certain that if the bots aren't currently doing a preliminary scan, it's only a matter of time before they will. Thus using a non-standard port is little more than a temporary side-stepping of the problem. Should you opt to try it, don't forget to open the new port in your firewall first or you'll lock yourself out of your server.

Secure Authentication - setting things up so that your computer authenticates itself to the server rather than logging in with a password - means you can then disable the ability to make connection by logging in with a password and this is very cool option so long as you don't find yourself away from home, needing to log into your server from another machine, and without your authentication keys on a memory stick.

Other solutions that I've seen use programs like fail2ban or denyhosts to scan logs and act upon what they find. Typically they'll scan the logs at a 20 minute intervals and block any IP address that has tried and failed to log in more than a specified number of times. As I see it, the problems with this type of approach is that you'll still see numerous attempts taking place because the IP address isn't blocked until the log scan takes place.

pam_abl, an addition to PAM, looks for repeated failures to log in and if a given IP address fails more than X times in Y seconds, it gets banned. This has a more immediate effect however the info I saw suggested that the bans were permanent and that's not what I want. Any defence mechanism that blocks IP addresses should also ensure that they are unblocked a short while later because if the attacker is logged into the web via an IP addess allocated from a pool at AOL or attacking via a hacked server, then you could end up blocking legitimate traffic when the address has been reallocated, or the server being sorted out. I guess using a cron job to unblock them would be an option.

Port knocking - a scheme by which you have to 'knock' on a number of prespecified ports in a set sequence in order to open another port - had an immediate appeal because the knock cab be carried out with something as commonly available as Telnet so getting access while away from your normal machine merely requires that you remember the port numbers. Wikipedia has a good article on port knocking with links to an article explaining how to do it using iptables (although you need to have the ipt_recent module loaded to do it).

However, while discussion port knocking over at the Fedora project forums (click here to see the topic), sej7278 suggested that you could use something on a webpage to trigger the opening an closing of a port. Further to that it occurred to me that you could use a url that was normally used for something else. For example, if the front page of my site were:

http://www.luntaticatics.com/index.php

I could use:

http://www.luntaticatics.com/index.php?open_port=22

to open SSH. All I'd need to do is to make the index.php script look for the additional parameters and use exec() to issue the command to open the port.

I'll explain how that works in a later post however for now I would just like to say to any crackers who may be reading this:

1. The front page of my site is not index.php and I'm not using a page at lunaticantics.com so you'll have to start by guessing the name an location of the .php file that I'm using. Note that there's no reason for the page to be linked into the rest of the site so you'll have to guess.

2. I wouldn't be so stupid as to use something as simple as open_port=22 and publish it here. So if you do manage to find the .php file, you'll need to guess what parameters are needed to trigger it. Note that there's no need for the parameter name and value to make any sense to anybody or anything other than myself and the .php script that's watching for it.

3. Probably your best bet for finding the above to monitor the network traffic to my server but even if you do find it, you will then need to figure out my SSH user login name, password, and root password (as described in my earlier post you can't just log in as root), and any failed attempts will be show up in my Logwatch report thus alerting me that you've found my port opener, and prompting me to change it.