Archive for linux

Network backup with netcat (nc) and tar

// October 3rd, 2009 // 1 Comment » // linux

Sometimes I need to recover some data from a server via network and either don’t have enough space on the source server I want to backup or just want to get things copied to an other server really quickly.

In this cases I usually just use the abilities of tar and netcat to create a backup via the network.

Let’s assume we have two computers:

- Computer A) which will be backed up.
- Computer B), which ip addess is our REMOTE_IP and the backup will be stored in this computer.

To create a backup we login to Computer B and launch netcat in listening mode using the following command.

nc -l -p 6969 >/root/backup.tgz

This command tells netcat to start listening on port 6969 (you can use other free ports) and sends whatever it receives to /root/backup.tgz.

Once done the following command, executed on computer A will start the backup:

sudo tar cvzp --same-owner --exclude=/root/backup.log --exclude=/proc/* --exclude=/media/* --exclude=/dev/* --exclude=/mnt/* --exclude=/sys/* --exclude=/tmp/* / 2>/root/backup.log | nc -w 3 REMOTE_IP 6969

This will create a full backup of your computer and excludes various system files which usually should not be copied. It also creates a logfile in /root/backup.log.

NOTES

  • make sure to replace REMOTE_IP with the ip address of computer B and that the port number 6969 matches the one on computer B.
  • sometimes you might not have netcat available as “nc” and you need to type “netcat” instead.

Get a list of urls/domains from a text file

// November 5th, 2008 // 2 Comments » // linux

I was just in need of a little script that extracts all urls from a text file. Here is the result.

sed 's/http/\^http/g' FILENAME | tr -s "^" "\n" | grep http| sed 's/[\ |\\\|\"].*//g' | sed "s/['].*//g" | sort | uniq

And as my final goal was to extract a list of domain names from the file which I can later use in my php script here is the hardcore version which gives you a copy&paste array definition of all domains found in a file.

echo -n '$domains = array ( "' ; sed 's/http/\^http/g' FILENAME | tr -s "^" "\n" | grep http| sed 's/[\ |\\\|\"].*//g' | sed "s/['].*//g" | sort | uniq | awk 'BEGIN{FS="/"}{print $3}' | cut -d . -f 2- | grep -E '^[a-z]+\.[a-z]+$' | sort | uniq | tr "\n" "," | sed "s/,/\", \"/ig" | sed 's/, \"$//ig'; echo -n ' );'

Fix Moveable Type Export with Metatag TAGS instead of KEYWORDS

// October 26th, 2008 // 2 Comments » // linux, wordpress

Sometimes a Moveable Type export puts the tags in a Metatag field called TAGS. Sadly this kind of export file does not preserve the keywords/tags when it is imported in Wordpress. Here is a little fix to get these TAGS converted to KEYWORDS.

cat original.txt | sed '/^TAGS/s/ /,/ig' | sed '/^TAGS/s/"//ig' > converted1.txt ;let lines=`cat converted1.txt|wc -l`; let lines=`expr "$lines" : '\([0-9]*\)'`; for i in `seq 1 $lines`; do line=`head -n $i converted1.txt | tail -n 1`; istag=`echo $line|grep -E "^TAGS:"|wc -c`; if [ "$istag" -gt 0 ]; then tags=`echo $line|sed 's/TAGS: \\(.*\\)/\\1/g'`; echo $tags; fi; iskeyword=`echo $line|grep -E "^KEYWORDS:"|wc -c`; if [ "$iskeyword" -gt 0 ]; then let tagline=($i+1); fi; if [ "$i" == "$tagline" ]; then echo $tags; tags=""; tagline=0; else echo $line; fi; done > converted.txt

Simple way to get a diff to a server without subversion

// October 26th, 2008 // No Comments » // linux

I recently ran into the problem that I had to update the changes I made on my subversion sandbox to a shared hosting account.

In general you would just check out the revision you need or run a svn diff.

Sadly this is not only possible and you need to figure out the changed files which you need to copy to the server.

This simple little command creates a tar archive with the changed files.

Note: you should be in the path of the checked out repository.

let revision_start=196; let revision_end=225; tar cfv ../r$revision_start-$revision_end.tar `svn diff -r $revision_start:$revision_end . | grep Index | awk '{print \$2}'`

lighttpd for image hosting

// October 10th, 2008 // No Comments » // linux

In order to take some work of the webserver I recently installed a lighttpd server for delivery of static files.

The following tuning did the job (thanks to the video sharing script)

/etc/lighttpd/lighttpd.conf

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.event-handler = "linux-sysepoll"
server.network-backend = "linux-sendfile"
server.max-fds = 8192
server.stat-cache-engine = "fam"

/etc/sysctl.conf

# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
net.core.wmem_max = 1048576
net.ipv4.tcp_wmem = 4096 87380 524288

in the apache virtual host we just use proxy pass to forward our static urls to the lighttpd server running on port 8080

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /images http://127.0.0.1:8080/images
ProxyPass /js http://127.0.0.1:8080/js
ProxyPass /css http://127.0.0.1:8080/css
ProxyPassReverse / http://127.0.0.1:8080/