After reading depesz’s great speeding-up-dumprestore-process blog post, I started thinking about how to securely transfer a file from one server to another in the fastest possible way. The problem being that scp/sftp is slow for various reasons. ftp, http, nc, cifs, rsync are all plain text so can be quickly discounted. I don’t know if ssltunnel suffers from the same window limitations that ssh suffers from. I guess using a dedicated VPN would do the trick. However I liked the idea of using gpg. I was not sure which cipher to use so I decided to run a few benchmarks to see. The results are on my site. Once the encrypted file has been created then it can be transferred using any of the available plain text mechanisms. I think nc or ftp have the least overhead.
So today I played a bit with some big cheap disks. I have 3 fairly old desktops each with 4*1TB disks, all exported via ATA over Ethernet to a much more modern sort of disk head. Basically it’s a “build a large store on half a shoe string” project. I’ve not quite got the network side of it sorted yet. Currently the disk head’s gigabit card is being saturated. On a single disk node each disk can do about 80M read sustained. If all 4 are read at the same time it goes down to about 50M, which is 200M in total. Which seems quite amazing for a Pentium 4 3.0Ghz. Also this seems a bit high as the PCI bus can only do 133M, I’m guessing that the onboard sata ports are somehow separated from the extra pci sata card I added. Interestingly one disk node can sustain 80M read from disk to the disk head. Again this backs down to 30M if all 4 disks are read, that’s 120M total, no surprise this is saturating the gigabit link. So the major bottleneck is the disk head. Currently max raid sync speed is 10M, ie 120M total for 12 disks. Ideally 3 nics in the disk head would be best, but then there is no way for the data to get to the disk head.
So my gym is closed for 10 days for a refurbishment. They are going to replace all the cardio equipment with new kit. I’ll be able to check my progress online and I’ll be able to plug in a video iPod and watch on the screen. The problem is, while it’s closed I have no gym to go to! So I decided to try some classes instead. Tonight I tried a yoga class and it was great - both relaxing and tiring at the same time. I really enjoyed it and think I’ll be going more often in the future; hopefully it will help improve my skiing.
I went to Tony and Grainne’s wedding on Friday. It was a lovely day and I wish them all the best in the future.
I got back on Monday night after cutting it worryingly close to get to the airport; two hours in advance, try 40 min! Only for Ryanair to delay the flight by an hour and tell no one. Then there’s the Post Holiday Blues, aka PTD. It hit bad on Tuesday night but I think I’m used to the grindstone again now.
No Internet for 2 weeks, just as I catch up on my feeds too!
I just found out that Cineworld have a public XML data source. It’s over in the syndication subdir of their main site. There are more technical details over on my CineworldScrape page. I feel a bit sad in a way, my scraper has been working for more or less 3 years now in some form or another. The annoying thing is that they don’t actually export all the data I use. Being quite visual I have grown used to using the thumbnails that each movie has to help me pick which films to see. Apart from this it’s a great step forward. However, the problem is there’s no knowing if Cineworld will pull this service or even if they want it public. For now I’ll continue to use my scraper.
I passed my foundation exam on Sunday. Big thanks to all the helpful folk at SADARS. I am now M6TDS.
Today I found out where Linux exposes the extra routing information gathered from ICMP redirects. ip route show cache will show the entire cached routing table. It’s a bit hard to read so ip route show cache 1.2.3.4 is better. For example 192.168.1.0/24 is a network that is connected via a host on my 192.168.0/24 network. My default gateway (192.168.0.1) has a static routing entry to the host that gateways for the 192.168.1.0/24 network (192.168.0.57). So when a random host on the 192.168.0.0/24 network pings a host on the 192.168.1.0/24 network it first sends to 192.168.0.1 but it sends an ICMP redirect saying that in the future it would be better to just send direct to 192.168.0.57.
I learned over the years to read the old ide subsystem errors for Linux and generally am able to get a feel for the sort of hardware error that’s coming. However I have yet to really get a feel for the libata errors, I’m not really used to reading the errors. A friend linked me to a page on the libata wiki.
I played with Fedora more today. There are loads of packages in the default repos these days. However I still find myself missing things from Debian. For example, I can’t do a “yum search” while doing a “yum upgrade”. Why? I also miss the small helper scripts that save on legwork. For example update-grub: in Debian when a new kernel gets installed the initrd and grub get updated and it just works. With Fedora I have to manually update the grub config and if I forget to create the initrd, it means a trip to the 13 PCs I just upgraded!
I have finally documented CineworldScrape and made it available to the general public. So with any luck Google will index it!
