Goodies in BitTorrent specifications

While looking through BitTorrent specifications, I noticed two parameters which are sent by leecher/seeder to tracker:

uploaded
    The total amount uploaded so far, encoded in base ten ascii.
downloaded
    The total amount downloaded so far, encoded in base ten ascii.

Reminds me freedom of RCPT FROM: in SMTP (or Caller-Id: header in SIP) in a suitable context and sloppy service providers. :-)

2012-05-12 10:43:45-0700

erlmpc and awesomeness of PIQI

Piqi is the second most awesome thing in Erlang (after Proper).

Disclaimer: a “guy” in this post can be both woman and man. Just sounds better when spoken about programmer folk.

How it started

This year I sold my summer to Netherlands, those guys seem to be pretty cool. My start of work is approaching, and I asked for some homework in order to get into the problem domain and familiarize myself with the tools. However, didn’t get any (for whatever reason. Either didn’t want to bother, or I seem to be overqualified). So I had to think some by myself.

Since there is reasonably large infrastructure and there are many teams working on different things (database guys, UI guys, gamer guys, money guys…), it is necessary to separate concerns. Now I am most interested in separating gamer guys (game UI) from programmers (game backend). We don’t really want to know if they are using flash, HTML5 or whatever else. From their perspective, they don’t even want to know Erlang exists (in reality, they are using flash, and we are using Erlang). The transport layer is JSON.

A formal schema definition, a parser and validator for it are needed. In this case, plain and naive JSON is worse than XML, because it doesn’t have:

XSD
WSDL

Both these are horrible, but can do their job when asked nicely. And they are only for XML. We needed some data definition language and parser which would work for JSON as well, and would be capable to convert to/from JSON and XML.

I invented this in my previous job. Given a schema, it was able to convert from Erlang to JSON and XML and vice-versa. In my previous job we needed to support both XML (for “enterprise” clients who connect to our system via SOAP) and JSON for communication with browser. I implemented and tested the initial version in 3-4 weeks. While this Data Definition Language (DDL) had much love and time invested, was tested PropErly, it didn’t feel right. A home-brewn solution for such an important thing.

Getting to the truth

Before writing my DDL I was looking for something what is already there. Erlangers suggested piqi. I looked at it, liked it, but somehow thought that it needs OCaml at run-time. I thought: “an Erlang port running alongside just to (de)serialize our stuff? Operations will get infinitely more complicated, so this will never happen”. This is how piqi was crossed out.

However, when I was at the Dutch guys, they were amazed about piqi. For whatever unknown reason. So I thought to try it out myself.

Here is the gist of this post: piqi does not require OCaml at run-time. In fact, plain Erlang is sufficient. I will make an effort to get this this explicitly included into erlang-piqi documentation. I think, this gist could have saved me a lot of time.

A bit about piqi

It is a data definition language. In a nutshell, a formal agreement: “I will send you a record which will have one field: volume, and volume will be an integer”. I create a record in Erlang with data I want to send, and pass it to piqi. Piqi converts it to whatever (now whatever can be xml, json, Google protocol buffers, or piqi itself) and sends it over the wire.

If I get JSON from JavaScript, I pass it to piqi. Piqi validates it and converts to native Erlang type. The advantages are:

Data type validation. Is 5 a binary, integer, float or uint32?
Complex data types and escaping. I can trust output processed by piqi.

This is not a full overview of piqi features. It can do much more. I present only the angle of it I used it for my Sunday project.

Here is an example piqi data definition and corresponding generated .hrl file:

erlmpc.piqi erlmpc_piqi.hrl

erlmpc.piqi	erlmpc_piqi.hrl
.variant [ .name request .option [ .name setvol .type int ] .option [ .name seek .type int ] .option [ .name next ] .option [ .name prev ] .option [ .name pause .type bool ] .option [ .name status ] .option [ .name currentsong ] .option [ .name statuscurrentsong ] ]	`-type(erlmpc_request() :: {setvol, integer()} \| {seek, integer()} \| next \| prev \| {pause, boolean()} \| status \| currentsong \| statuscurrentsong ). -endif.`

    .variant [
        .name request
        .option [ .name setvol .type int ]
        .option [ .name seek .type int ]
        .option [ .name next ]
        .option [ .name prev ]
        .option [ .name pause .type bool ]
        .option [ .name status ]
        .option [ .name currentsong ]
        .option [ .name statuscurrentsong ]
    ]

      -type(erlmpc_request() :: 
            {setvol, integer()}
          | {seek, integer()}
          | next
          | prev
          | {pause, boolean()}
          | status
          | currentsong
          | statuscurrentsong
      ).
      -endif.

request defines a request from user to the music player. If you had a look at full files, there are some custom types defined, like:

.enum [
    .name state
    .option [ .name play ]
    .option [ .name stop ]
    .option [ .name pause ]
]

And they are used like first-class citizens like int or string. Piqi DDL is powerful.

All Erlang files are generated. You need OCaml and the whole stack only for generating .erl and .hrl files (this is why I included them to the repo. I don’t want to be rough and force erlmpc users to have OCaml just to run my music player interface).

I encourage using piqi if you have two systems that have to speak to each other. I was very impressed with its support for Erlang.

Other buzzwords

I knew mochiweb from the past pretty well, so wanted to try another buzzword for web server. Tried cowboy. Amazing tool. This is the way libraries/tools should be written. I will not get into big details here.

erlmpd is a very, very nicely written library as for Erlang R13 (now is R15B1). Documentation is great. Type specs, though optional at that time (and were visible only by edoc, since there were no good type checkers then), are very nice and helpful. This guy certainly knew what he is doing. It was trivial to convert his library (3 years old) to OTP application, add rebar and use straight away.

WebSockets. It would be too much hassle to implement server-browser communication via long-polling (the only way not using WebSockets to get a close to real time feeling). So I chose WebSockets and didn’t bother. And I am happy I did it. Ability to push from server makes things so much easier! What is more, you do not need to implement any session support. For every connection, you get State. Single connection is a single client. In this State variable I hold a connection to MPD server. No cookies, no manual routing user requests to processes, handling disconnects and etcetra. It just works.

Of course, if you are writing software for general public or organizations which use IE7, this will yet not work. But future is there.

Thoughts about the product

The initial version 0.1 took a long Sunday.

While small and limited, this the best web-based MPD client from set-up perspective. Looks like this:

Install Erlang
Get sources
make run
Profit!

No sudo make install, no web server configurations. Nothing! Enjoyable, isn’t it?

UI JavaScript part is the only ugly part in the whole codebase. For now, it’s understandable. But it has to be rewritten if we want to extend this client. If I will continue this project, I suspect Backbone.js will be a good candidate for that. We need proper pub/sub for UI. Otherwise, it’s ugly.

All in all, things learned:

how piqi works, from absolutely nothing to something usable
music player communicates with server through WebSockets (only)
cowboy was the chosen web server, which I was really happy with. I should dedicate another post for awesomeness of cowboy.

Oh, and here is the screenshot:

Version 0.1

Are you a UI designer, MPD user or Erlanger? Contributors are very welcome!

2012-04-30 00:39:00-0700

Converting UK NPTDR data to GTFS

Took sime time to convert Glasgow Bus TXC to GTFS. I will show the problem and wat to tackle it:

 % ./tXCh2GT.sh ../GLA/ATCO_609_BUS.txc UTC 1 /tmp ../GLA/Stops.csv 
 transxchange2GTFS 1.7.4
 Please refer to LICENSE file for licensing information
 Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 44, Size: 1
 ...

Here is a work-around:

 diff --git a/TransxchangeTrips.java b/TransxchangeTrips.java
 index e104bb4..eafadaa 100755
 --- a/java/Transxchange2GoogleTransit/src/transxchange2GoogleTransitHandler/TransxchangeTrips.java
 +++ b/java/Transxchange2GoogleTransit/src/transxchange2GoogleTransitHandler/TransxchangeTrips.java
 @@ -225,7 +225,8 @@ public class TransxchangeTrips extends TransxchangeDataAspect {
    if (found) {
        // matching service found
        // generate service key specifically for current VehicleJourney
 -      handler.getCalendar().calendarDuplicateService(_serviceCode, _serviceCode + "_" + _vehicleJourneyCode + "@" + _departureTime);
 +      //handler.getCalendar().calendarDuplicateService(_serviceCode, _serviceCode + "_" + _vehicleJourneyCode + "@" + _departureTime);^M
        _serviceCode = _serviceCode + "_" + _vehicleJourneyCode + "@" + _departureTime;
        handler.getCalendarDates().calendarDatesRolloutOOLDates(_serviceCode);
    }

Then you will need a lot of RAM (for Glasgow buses 208 GB is not enough). To tackle this, change all dates that are more than 5 years after now to whatever is now + 2. The problem is that some bus timetables are valid until 2099-12-31, which explains why it takes so much memory.

Lots of stops will not be there if you will use only Glasgow (region 609)… So download full UK stop data.

Then you will get some kind of google_transit.zip. And some tunings to the folder with txts’:

#!/bin/sh

set -ex

# Fill agency.txt
awk -F, '{print $2","$2",http://example.org/"$2",UTC,en,1234"}' routes.txt | sort -u >> agency.txt

# Some station names have commas, remove them
perl -i -lne '@a = split (",", $_); print $. == 1? $_ : join(",", @a[0..2], join("_", @a[3..($#a-1)]), @a[$#a..$#a+2])' trips.txt

# No calendar dates for any service.. Force it
sed -i 's/,0/,1/g' calendar.txt

# Some routes have too early ending date
sed -i 's/20091231/20131231/' calendar.txt

# Remove routing exceptions which disables routes
sed -i '/2$/d' calendar_dates.txt

# Regenerate stops.txt with proper lat/lng
awk -F, '{print $1","$7",,"$5","$6",,"}' Stops.csv | sed 's/"//g' > stops.txt
sed -i '1s/.*/stop_id,stop_name,stop_desc,stop_lon,stop_lat,zone_id,stop_url/' stops.txt

# Remove 0000SPT stop_times, since these stops are not in Stops.csv
sed -i '/0000SPT[0-9]/d' stop_times.txt

And now you have a google_transit.zip which can be fed to Open Trip Planner! Enjoy!

2012-03-17 07:35:00-0700

Xen saga, part 3

When building Xen-4.1 on Squeeze, xen-utils-4.1 has this dependency list:

  Depends: e2fslibs (>= 1.41.0), libc6 (>= 2.7), libgnutls26 (>= 2.7.14-0),
  libncurses5 (>= 5.7+20100313), libpci3 (>= 1:3.1.7), libuuid1 (>= 2.16),
  libxen-4.1 (>= 4.1.2), libxenstore3.0 (>= 4.0.1~rc4), zlib1g (>=
  1:1.1.4), python (>= 2.6.6-3+squeeze3~), python (<< 2.6),
  xen-utils-common (>= 3.4.2-4)

Seems to be related to #644573, however, I could not trace it and fix it properly. For testing I as a workaround I manually removed python (<< 2.6) dependency from xen-utils-4.1_4.1.2-2_amd64.deb. This rendered xen-utils-4.1 installable.

After successful xen-4.1 stack installation I fired up virt-install:

motiejus@skveez> virt-install --name=chef-server --nographics --paravirt
--ram=512 --location=/home/motiejus/stuff/install/debxen/
--disk=/home/motiejus/stuff/chefDemoServer.dat,size=10,format=raw

Starting install...
Retrieving file MANIFEST...
Retrieving file vmlinuz...
Retrieving file initrd.gz...
ERROR    POST operation failed: xend_post: error from xen daemon: (xend.err
    'Device 0 (vif) could not be connected. Hotplug scripts not working.')
    Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect xen:/// start chef-server

Same story with Xen-4.0. Neither solution, nor workaround. Further steps:

Try same thing on fresh Debian Testing installation (thanks chroot)
Use native Xen utils without libvirt for testing
Ping Xen mailing list

Update 2012-03-04 00:17:

Xen in chroot yields exactly the same result. However, quick run of xen-create-image (from xen-utils) yields a promising result. My suspision is Xen isn’t really libvirt friendly.

Part 4 will be about xen-utils.

P.S. initial KVM setup with Ubuntu Lucid took around 2 hours including:

Server installation
First virtual machine installation (virt-install) to an lvm volume
Simple networking configuration

All using libvirt, around 6 moths ago. Xen apparently needs a bit more fight, at least with *deb-based. I suppose it would be less hassle with CentOS/RHEL/Fedora; Citrix XenServer RHEL based, anyway.

2012-03-03 14:24:00-0800

Xen on Squeeze via libvirt, part 2

This is the continued attempt to run Xen-4 on squeeze via libvirt. Why libvirt? Because if we manage to stick to it, we will have an awesome interface in the future when we want to interfere with our hypervisors/VMs.

virt-install failed to do a very standard installation. One of the gotchas: xen-utils-common/squeeze is inconsistent:

/etc/xen/scripts/network-route calls hotplugpath.sh, which exists only in xen-utils-common/sid. Which, according to my suspisions, assumes Xen-4.1.

I did not dig really deeply into this, because it is not worth it. Xen-4.1 (which is in testing) has a different approach to configure networking. As well as many things like xl instead of xm.

Installing xen-hypervisor-4.1/testing offers to switch the whole system to testing. Not an option. We do not want to use Ubuntu, right? So I continue my efforts to backport Xen-4.1 to testing.

What next?

Port Xen-4.1 with the whole userspace to stable.

2012-02-29 06:56:00-0800

Xen on squeeze laptop

Motivation

This is the first series of adventures trying to run Xen on a stable Squeeze machine, doing as few work as possible. My final goal is:

bootstrap new VMs using virt-install
have some kind of networking (ideally, a bridge in the hypervisor so I can manually configure iptables masquerading)

As few work as possible – no compiling, as few as possible configuration file changes, no hacks. I am aiming for a clean installation, since will have to create a chef script for that.

Process

Since Linux 3.0 you do not need an other kernel, everything’s merged well enough. Just install xen and friends:

xen-hypervisor-4.0 xen-tools xen-utils-4.0 xenstore-utils virtinst xen-docs-4.0 libvirt-bin

Short configuration reference. Configure this line:

(xend-unix-server yes)

So we can use xen from libvirt. Default libvirt network suits fine, so change this as well:

(vif-script 'vif-bridge bridge=virbr0')

Download bootstrap images:

 wget -rl0 -np ftp://ftp.lt.debian.org/debian/dists/squeeze/main/installer-amd64/

And install:

virt-install --name=chef-server --file-size=10 --nographics --paravirt --ram=512 \
                    --file=/install/debian-6.0.4-amd64-businesscard.iso \
                    --location=/install/debxen/installer-amd64/

Hacks

Haha.

Install libvirt-bin from squeeze-backports while trying to work around this:

ERROR POST operation failed: xend_post: error from xen daemon: (xend.err ‘Device 0 (vif) could not be connected. Hotplug scripts not working.’)
then new libvirt mangles up the paths. mkdir /var/lib/xen/ because of

ERROR Could not start storage pool: cannot open path ‘/var/lib/xen’: No such file or directory
apt-get install -t sid xen-utils-common because of:

Feb 28 21:24:37 skveez udevd-work[6494]: ‘/etc/xen/scripts/block’ (stderr) ‘/etc/xen/scripts/xen-hotplug-common.sh: line 20: /etc/xen/scripts/hotplugpath.sh: No such file or directory’

After that I realized I was looking to the wrong direction. My syslog pointed me:

/etc/xen/scripts/block: Writing backend/vbd/11/51712/hotplug-error Path closed or removed during hotplug add: backend/vbd/11/51712 state: 1 backend/vbd/11/51712/hotplug-status error to xenstore.

Well.. quick googling found this. I do not want to upgrade to xen-4.1 why not?.

To be continued…

2012-02-28 11:41:00-0800

Most popular discussions in Erlang-questions

After a long thead about Misultin EOL in erlang-questions I wanted to check how active is it comparing to other threads in the mailing list. Honestly, I was expecting for it to be at least in top 5.. But I was wrong.

Here are the top 50 discussions as of of 2012 January 19 00:15 UTC:

  1 | 113 | 2008-11-15 - 2008-11-21 | [erlang-questions] Erlang 3000?
  2 | 108 | 2011-05-24 - 2011-05-26 | [erlang-questions] Why do we need modules at all?
  3 |  91 | 2008-03-10 - 2008-04-01 | [erlang-questions] erlang sucks
  4 |  85 | 2009-02-24 - 2009-03-05 | [erlang-questions] The Beauty of Erlang Syntax
  5 |  83 | 2009-03-17 - 2009-05-09 | [erlang-questions] Reassigning variables
  6 |  83 | 2007-05-21 - 2007-06-06 | [erlang-questions] some language changes
  7 |  81 | 2008-05-29 - 2008-06-10 | [erlang-questions] Twoorl: an open source Twitter clone
  8 |  77 | 2005-07-11 - 2005-08-01 | Meyer, OO and concurrency
  9 |  76 | 2006-02-13 - 2006-03-02 | Longstanding issues: structs & standalone Erlang
 10 |  75 | 2003-12-10 - 2004-01-14 | Small poll
 11 |  75 | 2006-10-30 - 2006-11-10 | [erlang-questions] Package Support/Use
 12 |  71 | 2008-02-12 - 2008-03-01 | [erlang-questions] Strings as Lists
 13 |  67 | 2003-01-27 - 2003-02-05 | A Joeish Erlang distribution (long)
 14 |  66 | 2008-06-24 - 2008-08-05 | [erlang-questions] Ideas for a new Erlang
 15 |  65 | 2004-01-12 - 2004-01-28 | ANNOUNCE - graphics package
 16 |  64 | 2006-08-03 - 2006-08-15 | Mac Intel
 17 |  64 | 2011-07-26 - 2011-08-01 | [erlang-questions] trouble with erlang or erlang is a ghetto
 18 |  63 | 2011-02-12 - 2011-03-11 | [erlang-questions] Two beautiful programs - or web programming
 19 |  60 | 2006-02-26 - 2006-03-13 | optimization of list comprehensions
 20 |  57 | 2011-06-12 - 2011-06-17 | [erlang-questions] A PropEr announcement
 21 |  53 | 2011-01-04 - 2011-01-17 | [erlang-questions] Erlang and the learning curve
 22 |  53 | 2011-07-21 - 2011-09-13 | [erlang-questions] Funargs: Ruby-like blocks for Erlang
 23 |  53 | 2003-04-01 - 2003-04-04 | FAQ terminology harmonisation
 24 |  51 | 2001-03-05 - 2001-03-12 | Erlang Development Environment
 25 |  51 | 2009-07-15 - 2009-07-23 | [erlang-questions] Re: Unicast 20k messages, $500-$1000 bounty
 26 |  51 | 2007-11-13 - 2007-11-16 | [erlang-questions] idea: service pack one
 27 |  51 | 2008-09-02 - 2008-09-05 | [erlang-questions] Parallel Shootout & a style question
 28 |  51 | 2003-10-22 - 2003-11-05 | Language change proposal
 29 |  50 | 2005-08-25 - 2005-09-09 | (Prolog + LISP + Erlang) with integration issues versus C++
 30 |  50 | 2011-05-18 - 2011-05-20 | [erlang-questions] DRY principle and the syntax inconsistency
 31 |  50 | 2010-10-13 - 2010-11-05 | [erlang-questions] Re: Shared/Hybrid Heap
 32 |  49 | 2010-11-13 - 2010-11-17 | [erlang-questions] Erlang shows its slow face!
 33 |  48 | 2007-11-26 - 2007-12-12 | [erlang-questions] benchmarks game harsh criticism
 34 |  48 | 2006-12-11 - 2006-12-22 | [erlang-questions] Why is Erlang what it is?
 35 |  47 | 2008-05-14 - 2008-06-04 | [erlang-questions] eep: multiple patterns
 36 |  47 | 2002-04-15 - 2002-04-22 | Erlang language issues
 37 |  47 | 2007-09-03 - 2007-09-10 | [erlang-questions] Intel Quad CPUs
 38 |  46 | 2011-10-17 - 2011-10-25 | [erlang-questions] Erlang newbie questions
 39 |  46 | 2006-06-26 - 2006-07-06 | Strings (was: Re: are Mnesia tables immutable?)
 40 |  46 | 2003-02-14 - 2003-02-19 | Musings on an Erlang GUI System.
 41 |  46 | 2003-09-18 - 2003-10-16 | Enhanced type guard syntax]
 42 |  46 | 2008-10-21 - 2008-11-05 | [erlang-questions] Why isn't erlang strongly typed?
 43 |  46 | 2003-10-13 - 2003-10-27 | Erlang is getting too big
 44 |  46 | 2009-11-08 - 2009-11-12 | [erlang-questions] Why Beam.smp crashes when memory is over?
 45 |  45 | 2004-03-21 - 2004-05-03 | user-defined operators
 46 |  45 | 2006-06-06 - 2006-06-13 | Language Bindings for Erlang Again (Opinion)
 47 |  44 | 2008-02-28 - 2008-03-11 | [erlang-questions] Use of makefiles
 48 |  44 | 2012-02-16 - 2012-02-18 | [erlang-questions] Misultin EOL
 49 |  44 | 2006-07-31 - 2006-08-09 | pcre, bifs, drivers and ports
 50 |  44 | 2008-09-12 - 2008-09-19 | [erlang-questions] My frustration with Erlang

Ah.. Only 48’th.. Guess I will have to wait for another millenium.

Script

2012-02-18 16:15:16-0800

The absolute minimum to bootstrap Jekyll

Motivation

So I wanted a Jekyll powered note engine, and I wanted to bootstrap quickly. While Jekyll Bootstrap fits for most, I am more like an LFS-guy. Do things in small steps from the start understanding what is going on.

I assume you have read the introduction and know what Jekyll is. This post will help you to bootstrap the bare minimal system.

Setup

So install it first. Then create some self-explanatory files:

_layouts/base.html _layouts/page.html

_layouts/base.html	_layouts/page.html
`<!DOCTYPE html> <html> <head><title>{{ page.title }}</title></head> <body>{{ content }}</body> </html>`	`--- layout: base --- <h3><a href="/">{{ page.title }}</a></h3> <div id="content">{{ content }}</div>`
index.html	_config.yml
`--- layout: base --- {% for post in site.posts %} <h3><a href="{{ post.url }}"> {{ post.title }}</a></h3> {{ post.content }} <hr/> {% endfor %}`	`permalink: /:year/:month/:title`

<!DOCTYPE html>
<html>
  <head><title>{{ page.title }}</title></head>
  <body>{{ content }}</body>
</html>

---
layout: base
---
<h3><a href="/">{{ page.title }}</a></h3>
<div id="content">{{ content }}</div>

index.html _config.yml

---
layout: base
---
{% for post in site.posts  %}
  <h3><a href="{{ post.url }}"> {{ post.title }}</a></h3>
  {{ post.content }}
  <hr/>
{% endfor %}

permalink: /:year/:month/:title

Add some content

_posts/2012-02-13-playing-around.markdown	_posts/2012-02-12-hello-jekyll.markdown
`--- layout: page title: Playing Around --- To see if forloop really works`	`--- layout: page title: Hello, Jekyll! --- Hello, Jekyll!`

Done

jekyll --server
firefox http://127.0.0.1:4000/

These are the basics which you can have at hand simply working, and go from here. Next things you would like to do:

Create /css/base.css and link from index.html.
Read the introduction and wiki while already understanding of what’s going on.

2012-02-13 11:38:30-0800

Do not listen RADVD on certain interfaces

University of Glasgow Computing Science department broadcasts broken ipv6 default route to LAN. Since the host has a correct ipv6 address and route is acquired from OpenVPN server, disabling listening of radvd requests on eth0 is needed.

Easiest solution that works is using ip6tables (reference):

ip6tables -A INPUT -d ff02::1/128 -i eth0 -p ipv6-icmp -j DROP.

A note from Paweł:

I usually do it this way:

echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_ra

(and set it up in sysctl.conf)

Thanks, Paweł, much cleaner.

2012-01-31 11:28:00-0800

Installing WikiDroid on Nook Touch

Offline wikipedia for Nook Touch was supposed to be my personal project for Embedded Systems 3 course. However, one evening I started investigation, and installed WikiDroyd successfully. How:

Upgrade Nook Touch to 1.1
TouchNooter 1.11.20 nooter
Replace boot loader with wifi one, so you can access ADB
AppMarket does not seem to work. So download and install wikidroyd manually . Version 1.4.10 worked for me.
Create the following file structure in /sdcard:

/ # find /sdcard/WikiDroyd
/sdcard/WikiDroyd
/sdcard/WikiDroyd/en
/sdcard/WikiDroyd/en/file_infos.txt
/sdcard/WikiDroyd/en/lt.whole.wikidroyd
/sdcard/WikiDroyd/file_infos.txt
/sdcard/WikiDroyd/tmp

file_infos.txt:

filename,timestamp,size,md5sum,shorturl
ar.whole.wikidroyd,1288738800,170949632,65f0c107c03050c1e0c1c38324752316,http://books2.wikidroyd.com/bdkit8
ca.whole.wikidroyd,1288476000,385560576,e3305a0e704924f2efd5b639b5bf51c3,http://books3.wikidroyd.com/eb6xah
cs.whole.wikidroyd,1289257200,294767616,93f65728c69f518c74f74a6bf6a15020,http://books2.wikidroyd.com/a74qi6
da.whole.wikidroyd,1288652400,156098560,bfbff181de7260699fff2fe08e6ffe35,http://books2.wikidroyd.com/xib5zh
de.10%_most_read.wikidroyd,1286920800,501722112,25eac3a809e521734b4ea125aeabaf8b,http://books2.wikidroyd.com/d7i4uh
de.whole.wikidroyd,1286920800,2127689728,3871f4ad08f4c9a0b17ac4cc4323de37,http://books2.wikidroyd.com/e9zpz6
en.10%_most_read.wikidroyd,1286748000,1848722432,0798642b8ea8491512eb4e5cb550b63b,http://books2.wikidroyd.com/spixs5
en.whole-1-3.wikidroyd,1276418676,2118196224,8f5525cd0eb8fc1ae4a60991ac717160,http://books3.wikidroyd.com/r8ifkm
en.whole-2-3.wikidroyd,1276418741,2118159360,b9a74d5efdfdacbdcf2953783e3c5a6e,http://books3.wikidroyd.com/nkg3ki
en.whole-3-3.wikidroyd,1276418809,2109579264,59b6da9872cb0c6a2b207b9112544d29,http://books3.wikidroyd.com/udt7g7
eo.whole.wikidroyd,1289257200,118763520,4af3a711ff90cd5c6a7383cf7d446c79,http://books2.wikidroyd.com/td5s5z
es.whole.wikidroyd,1288911600,1197604864,e126d0d0bc4b60443fb4b0a934669333,http://books3.wikidroyd.com/84dmgq
fi.whole.wikidroyd,1288738800,321738752,6128fac490213d212962446dd3e5303d,http://books2.wikidroyd.com/9sa2me
fr.whole.wikidroyd,1288738800,1871736832,50842968eb50fc9422ab55c542b15ea1,http://books3.wikidroyd.com/ywyxcb
he.whole.wikidroyd,1289084400,225228800,1027eef844c9e37b12029463dcb1849f,http://books2.wikidroyd.com/kn5mby
hu.whole.wikidroyd,1289084400,374478848,076efcf2477d2a8514f6ad597f815e5c,http://books2.wikidroyd.com/4qcznp
id.whole.wikidroyd,1289084400,152918016,093c3fc61c31ca31d4f076c75ca7c7b5,http://books2.wikidroyd.com/g5r5h6
it.whole.wikidroyd,1288738800,1369921536,4641b751bd26c40b814687f3d2f437af,http://books3.wikidroyd.com/zcdr45
ja.whole.wikidroyd,1288652400,1751064576,733f48516651c8caca727deb7520e7a0,http://books3.wikidroyd.com/w89ep2
ko.whole.wikidroyd,1288825200,217603072,9a87c862ce963b7aa7e7d019d5427848,http://books2.wikidroyd.com/enrykj
lt.whole.wikidroyd,1289084400,113582080,b3d93b49196c2ad0f3eb2667a4ce9de3,http://books2.wikidroyd.com/heu5zc
ms.whole.wikidroyd,1289084400,72553472,addd870fec4837b8abff8d2edd625eea,http://books2.wikidroyd.com/8gx6ye
nl.whole.wikidroyd,1288566000,693029888,d45cf37d8186ae7fc43bd48ce98aca6f,http://books2.wikidroyd.com/pf4mg5
no.whole.wikidroyd,1288476000,295270400,be2f93d7916fe56853a4cfa3f2659631,http://books2.wikidroyd.com/84idwf
pl.whole.wikidroyd,1288566000,948410368,d50c749fa1bd464f4850556b4b59bb96,http://books3.wikidroyd.com/a2yqn4
pt.whole.wikidroyd,1288476000,748855296,68f785fca79118c45e8676be7302ed83,http://books3.wikidroyd.com/vun93z
ro.whole.wikidroyd,1289170800,168313856,23327504ee66f0adadc6da7483da727e,http://books2.wikidroyd.com/apqjyx
ru.whole.wikidroyd,1287784800,1308120064,0d4e57e6c4c294f5abe418b389a61bc3,http://books3.wikidroyd.com/kkhzkf
simple.whole.wikidroyd,1288825200,57327616,6b4c923e8b74e2303983589887859667,http://books2.wikidroyd.com/piw9yi
sk.whole.wikidroyd,1288566000,121936896,1bc3046508f2e04d2f47fd0647b9be5b,http://books2.wikidroyd.com/vun37s
sr.whole.wikidroyd,1288652400,168373248,a4f46190031f3b744de3c52b74d3fb6d,http://books2.wikidroyd.com/iu6n7r
sv.whole.wikidroyd,1288476000,395089920,2db36eeef2b3497589a6ca98bb66593c,http://books3.wikidroyd.com/ymkpxs
tr.whole.wikidroyd,1289084400,216014848,696a81e096f4b65e1d0782a9bd317ed3,http://books2.wikidroyd.com/7a3u6a
uk.whole.wikidroyd,1288825200,358979584,4b8dcaf7eb52f0488218f01816c8a9fe,http://books2.wikidroyd.com/qtcxa7
vi.whole.wikidroyd,1288476000,168049664,9dd572392fd001e9bfbb30e16a17cd9f,http://books2.wikidroyd.com/z464d4
zh.whole.wikidroyd,1289257200,510788608,3209de683ed79bb15aca6d7ab4f3b19a,http://books2.wikidroyd.com/nz9zrw

I would suggest to install WikiDroyd on a phone, do the initialization there, and copy the files to the sdcard.

Changing any setting seems to crash WikiDroyd for the future. Reinstalling the application makes it work:

adb uninstall com.osa.android.wikidroyd

How to change font size without crashing the app for ever is left as an easy exercise for the reader. Hint: you might want busybox. :)

2012-01-12 09:45:00-0800

Ad-hoc switching keyboard layouts

When:

GNOME or something else does not allow you to switch keyboard layout in Xorg.conf (or udev) way
You have 2 or more different keyboard layouts

Insert to some keyboard shortcut:

setxkbmap -print | perl -ne 'print $1 eq "lt" ? "us" : "lt" if $_ =~ /\+(lt|us)\+/' | xargs setxkbmap

Oh, and this helps on Debian based systems (tested on squeeze):

$ cat /etc/default/keyboard 
# Check /usr/share/doc/keyboard-configuration/README.Debian for
# documentation on what to do after having modified this file.

# The following variables describe your keyboard and can have the same
# values as the XkbModel, XkbLayout, XkbVariant and XkbOptions options
# in /etc/X11/xorg.conf.

XKBMODEL="pc105"
XKBLAYOUT="lt,lt"
XKBVARIANT="us,"
XKBOPTIONS="grp:alt_shift_toggle,grp_led:scroll"

# If you don't want to use the XKB layout on the console, you can
# specify an alternative keymap.  Make sure it will be accessible
# before /usr is mounted.
# KMAP=/etc/console-setup/defkeymap.kmap.gz

2010-09-18 09:46:00-0700

Auto login using keycom

I live in Murano Street Student Village (MSSV). Internet is great here, but, unfortunately, there is one annoying thing: every time you turn on your computer, in order to have internet, you have to:

Open up your browser
Take your mouse (or move a finger to touchpad)
Aim to the “log in” button
Press the button

It’s too much for me.

This is how to avoid it. Create an executable script in /etc/network/if-up.d/ (or something like that) with contents like this:

#!/bin/sh
USERNAME="your_username" # urlencoded
PASSWORD="your_password" # urlencoded
if [ "$IP4_NAMESERVERS" = "109.246.240.1" ]; then
  curl http://login.keycom.co.uk:8080/goform/HtmlLoginRequest \
    -d"login=Sign%20in&amp;password=${PASSWORD}&amp;username=${USERNAME}" 2>&1 | logger
fi

If it doesn’t work, check your DNS server and adjust the script. See /var/log/messages for more information.

2010-09-16 19:45:00-0700