|
| Greg's diary |
| August 2006 |
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Tuesday, 1 August 2006 | Echunga | |
| Top of page | ||
| next day | ||
| last day |
Off to an early start this morning, which was just as well: discovered that I had a surprising amount of work to do. Part of that came to light in a two-hour phone call with Lars, who is currently in San Francisco.
Merging things with bitkeeper is a pain! Spent a lot of time trying to do a clean merge for BUG#13926, and ultimately failed. I suppose it's OK if you're prepared to commit separate patches showing all the mess I made trying to get there, but I'd rather do one patch per operation, and that's made more difficult by the fact that you need to commit individual components; if you then discover that something was wrong, you have to back out the commit. Sometimes it seems easier just to start again.
In the afternoon, phone call from Tim Gaden of APC magazine, who wanted an interview for their column “Extreme Geek”. Had quite a lively discussion; it'll be interesting to see which parts he chooses for his column.
| Wednesday, 2 August 2006 | Echunga | Images for 2 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Yvonne has not had a good year this year. On 2 January a horse fell on her and knocked her ribs about, on 5 June a horse banged her head against the bar at the front of a float, and today one paniced and kicked her, doing her shin no good.
Another long day today, a lot of it spent thinking about BUG#11312:
One of the tackiest things about SQL syntax is that stored procedures use the same delimiter for statements in stored procedures as in other statements, and that the syntax doesn't make it easy to distinguish the scope of the stored procedure. The “solution” (which I'd consider a poor workaround) is the DELIMITER keyword, which changes the character (sequence) that the client recognizes to represent the end-of-statement delimiter. From the manual, a typical stored procedure definition might be:
DELIMITER // CREATE PROCEDURE simpleproc (OUT param1 INT) BEGIN SELECT COUNT(*) INTO param1 FROM t; END; // DELIMITER ;
The problem in BUG#11312 is that we try to replicate this stuff. But the DELIMITER kludge^Wcommand is a client-side thing which never makes it to the server, so we can't replicate it. The result has been that the mysql client can't handle the output of the mysqlbinlog utility. The same considerations apply to stored functions.
That's an easy enough problem to understand. But how do we fix it? Fortunately stored procedures are stored as a single entry in the binlog, but we still have to recognize them. And we certainly can't guess what delimiter the original user chose; but that doesn't seem important. In almost every example I've seen, it's been either // or $$, both of which have the advantage of not being valid delimiters under normal circumstances.
So when printing the binlog contents I can proceed by examining every binlog entry for a stored procedure or function, and if I find one, enclose it in DELIMITER // and DELIMITER ;. That's simple enough, but it reeks of kludge, and I'm left wondering if I haven't missed something. And of course, if I have missed something, I won't test for that something either, which would leave some poor unsuspecting user to trip over it. In addition, it really wants me to parse every entry, which is non-trivial in terms of time. I think it should be enough to do a poor man's lexical analysis.
| Thursday, 3 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More work on bugs today, and got the code for BUG#11312 finished—almost. To identify keywords, I have to perform a number of string comparisons. But SQL is not case-sensitive, so CREATE and create have the same meaning. That's simple: there's a function called strcasecmp() which compares independently of case. But that will only work if the strings are the same length, so I can't just pick the name out of a buffer and compare it. Never mind, that's what strncasecmp() is for: it compares up to a specific length. So this code will recognize CREATE and create (and also, for example, CreAte) in the buffer:
if (strncasecmp (buf, "create", strlen ("create") == 0)
{ ...
But MySQL supports multiple character sets, so we don't use the base string comparison functions: instead we have functions like my_strcasecmp(), which has a third parameter, specifying the character set. But which character set do we use? I'll find out, but it needs more digging than writing the code in the first place. In addition, we have my_strcasecmp(), but there's no my_strncasecmp(). What do we do there? Add a my_strncasecmp() just for this one piece of code? Or capitulate and use the locale-specific comparison, which would probably work. It's amazing how the little things take up so much time.
| Friday, 4 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More work on bugs today, also some mailing list stuff. Finished and committed my patch for BUG#11312, but also sent out a message to the developers asking for alternative suggestions:
Date: Fri, 4 Aug 2006 10:10:29 +0200
From: "Greg 'groggy' Lehey" <grog@mysql.com>
Subject: Thoughts on DELIMITER in mysqbinlog
BUG#11312 describes problems handling stored procedures in the output
of mysqlbinlog. The DELIMITER kludge is handled entirely in the
client, so it never makes it to the binlog. We have a patch, but we
don't like it: it involves connecting to the server to perform the
dump, and it's very intrusive.
I've done some examination and discussed with Lars, and we've
identified two solutions of varying ugliness:
1. When processing a stored procedure or function, generate DELIMITER
statements and log them to the binlog.
2. When printing out log events, recognize stored procedure and
function declarations and generate DELIMITER statements to
surround the declaration.
Version (1) has the advantage that we know exactly when to log the
DELIMITER command, but several disadvantages:
- It creates unnecessary binlog bloat. I'd guess that we can live
with this.
- It requires the slave to process the definition, which again means:
- Older slaves won't be compatible.
- We need to modify the parser.
Alternatively we could make a special log type that the slave
doesn't process, but this too wouldn't be backward compatible.
Version (2) has the advantage that we don't have any slave
compatibility problems; it's entirely in the mysqlbinlog program
(though the code itself is in sql/log_event.cc). On the other hand,
it requires parsing the events to recognize a stored procedure or
function.
Neither of these options are overly attractive. On the whole we've
decided that (2) is the lesser evil, but it's possible that we've
missed a better alternative. That's the main reason for this
message: can you think of a better one?
If not, I'd still like some input on variant 2:
- Should I write a "poor man's" parser to recognize the stored
procedures, or should I use something existing? I can't imagine
that anything existing would suit very well, so I've decided on the
"poor man's" version.
- What character set semantics should I use? From what I can see, all
the character set handling is in the server, and it's disabled when
MYSQL_CLIENT is set. As a result, I've decided on the standard
system-supplied functions. But it's possible I'm missing something
here.
If you're interested in more details, the patch to log_event.cc is at
http://lists.mysql.com/commits/10036.
Just before I went to bed, Sergei came up with a potentially better one: put a flag on the binlog entry. This is a twist on my variant (1), and existing mysqlbinlog programs won't trip over it, but on reflection it means that it won't be any use for existing binlog files. We still need to discuss it.
| Saturday, 5 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Brew day again today, and got off to a slow start. It wasn't as painful as it has been, but it still took up pretty much the whole day.
Wikipedia has come up with a new section on cookery, apparently not a word used much in American, where they use the word “cooking”. Spent some time looking around; it's not currently up to scratch with the other sections, and it's far too US-centric (currently the section on cooking beef states that anything less than “medium rare” is not recommended). Also, of course, all the temperatures are in degrees Fahrenheit. I need to do something there.
| Sunday, 6 August 2006 | Echunga | Images for 6 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Today finally found both the time and the courage to look at the question of recording video from my DVB-T card. As I had mentioned last month, tzap offers me the opportunity to record without further pain installing undocumented software. But it would be nice to have a wrapper program to control tuning, start and stop. So I wrote a little program called record. Most of the code is, not surprisingly, in the time routines—why is UNIX timekeeping such a mess? Ended up writing about 800 lines, very little of which had to do with actually starting and stopping the tzap process. And it worked. Well, almost. Stopping the process on occasion caused the hardware to get confused:
Aug 6 17:38:38 ceeveear kernel: cx88[0]/2: queue is empty - first active Aug 6 17:38:38 ceeveear kernel: cx88[0]/2: cx8802_start_dma w: 0, h: 0, f: 2 Aug 6 17:38:38 ceeveear kernel: cx88[0]/2: setting the interrupt mask Aug 6 17:38:38 ceeveear kernel: cx88[0]/2: [f3cbc0c0/0] cx8802_buf_queue - first active Aug 6 17:39:09 ceeveear kernel: cx88[0]/2: cx8802_restart_queue Aug 6 17:39:09 ceeveear kernel: cx88[0]/2: cx8802_restart_queue: queue is empty
This wasn't the first time; the log files showed that it had happened before in March. I didn't find a way round that beyond rebooting; hopefully it won't happen if I don't kill the tzap process.
In addition, the recording stopped after 2 GB:
00:35:24 of 01:00:00 (169%) 2047 MB testfoo File size limit exceeded
Why? There's nothing in this file system that has a 2 GB sensitivity. Tried looking in the man pages:
=== root@ceeveear (/dev/pts/0) /var/log 46 -> man 2 openNo manual entry for open in section 2 See 'man 7 undocumented' for help when manual pages are not available.=== root@ceeveear (/dev/pts/0) /var/log 47 ->
I could scream! Why do Debian consider documentation optional? On eucla, my Fedora machine, I found:
O_LARGEFILE
(LFS) Allow files whose sizes cannot be represented in an off_t
(but can be represented in an off64_t) to be opened.
But that's only for LFS, and a test confirmed that the flag wasn't even known.
Still, despite all, the day's work was a success. It's a sad recognition that it's easier to write your own program than install somebody else's. Documentation standards seem to have reached an all-time low.
| Monday, 7 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Monday, bloody Monday! Nothing seems to get done. At least I've made up for it by working less time; instead, as last week, I worked longer hours the rest of the week, which I think was more productive.
Chasing up the failure of record to write beyond 2 GB was interesting. First, I found the Debian development man pages and installed them. man 2 open still gave me nothing; the man pages was installed as /usr/share/man/man3/open.3posix.gz, which leaves me wondering if it's even the right man page. It says nothing about O_LARGEFILE (not surprising; it has nothing to do with POSIX.1). Looking for the flags in the required header files sys/stat.h and fcntl.h brought nothing: the header files are part of the incredible mess that is the Linux system header files. The actual definitions are hidden in a maze of #ifdefs that ultimately include /usr/include/asm/stat.h, which defines three incompatible structs, __old_kernel_stat, stat and stat64. struct stat has offsets that are unsigned long (i.e. 32 bits on i386), so that can't work.
Spent some time talking with Linux people, and finally came up with the fact that I needed to define _GNU_SOURCE to get the correct definitions, and then add O_LARGEFILE after all, and replace calls to fstat with calls to fstat64, and define the receiving structure as struct stat64. And then it finally worked. But what a mess! I don't think even the most convoluted System V headers were this bad.
More importantly, though, the man pages say nothing about this mess, neither a definition of the structures, nor a mention of the _GNU_SOURCE requirement, nor the restrictions on file size. By contrast FreeBSD is a model of cleanliness. About the only excuse I can find for Linux (and a feeble one at that) is that this is the price of compatibility, exacerbated in Linux' case by the fact that the kernel and userland come from different places.
| Tuesday, 8 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Mail from James Andrewartha today, pointing out the mistake that I had made yesterday: I had installed the wrong Debian man pages (manpages-posix-dev instead of manpages-dev). Under those circumstances, it's not surprising that they're in section 3 and don't refer to Linux specifics. But it's easy to make this mistake: apt-cache search manpages gives me 65 different packages, and it's easy to miss this:
manpages-de - German manpages manpages-de-dev - German development manpages manpages-dev - Manual pages about using GNU/Linux for development manpages-es - Spanish man pages
The real issue is that man pages shouldn't be something you have to go out and get. If you install development software, the man pages should be there too. That was one of the great new ideas with UNIX over 30 years ago. In those days, the man pages took up a significant amount of the available disk space, but people obviously thought it was worth it; nowadays it's just noise.
The manpages-dev pages appear identical to the man pages on my Fedora system, as you'd hope. As I've noted, they mention O_LARGEFILE, but nothing else. James also found a reference to LFS on Andreas Jaeger's web pages. It seems that LFS stands for Large File Support, not Log-structured File System. Somehow it's all not very satisfying: first, there's no need for this stuff, and secondly if it's needed, it should be documented in the system, not somebody's web site.
Today is census day, and punctually a census helper arrived to deliver the census forms by hiding them under the doormat—clearly she wasn't able to wait the 20 seconds that it takes me to get to the door.
Looking at the forms, which require careful printing in ALL CAPS in predetermined boxes, it's good that it can also be done online. I was given a sealed envelope of the kind often used for salary printouts, containing a 12-digit number and requiring the census form number (which is different). The web page itself has useful information like Please do not use the browser
The address had to be a residential address, but I was asked for a post code, which is associated with a post office. Since I have a choice of post office (Echunga 5153 or Meadows 5201) this is clearly nonsense, but the form wouldn't let me continue without “a valid post code”. It accepted 0000, though.
Navigation made difficult by having to use the scroll bars; PageUp/PageDown don't work unless you first click away from an input field.
While answering questions, I was asked questions with potential answers like “Same as in question 8”. Where's question 8? No idea. It took a while to realize that the overflowing text boxes had a number written at the end, where it was barely legible. That means that question 8 much have been on the previous form. How do I get there? I don't. So how can I answer the question? I guessed. Later it occurred to me that the questions in the printed form had the same number, but by that time I had finished.
Interestingly, there were no questions about Yvonne. Looking at the paper form, this is obviously a deficiency of the web site. At the beginning I specified that two people live here, but it didn't follow through. Possibly one of the error routines forgot her again. It looks as if I have submitted an invalid census statement. Wouldn't it have made sense to provide the possibility of reviewing the form before submitting?
| Wednesday, 9 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Today's phone call from Lars was a little different: things are changing at MySQL, and there's no place for me now. In more American than Swedish fashion, I'm now a free man again. I can't say I'm happy; the time at MySQL was very enjoyable and fulfilling. It's a pity that it didn't work out.
Spent my new-found freedom by playing around with record, the DVB recording program I wrote on Sunday, making it more user-friendly (for command line users, of course). One of the interesting things is that Linux doesn't have a setproctitle() function, which changes the text displayed by ps(1). In the end, found a way to fake one by overwriting the argv and envp parameters at the top of the stack:
int main (int argc, char *argv [], char *envp [])
...
for (i = 0;; i++)
{
if (envp [i])
env = envp [i];
else
break;
}
statusline = argv [0]; /* display in ps output */
statuslength = env + strlen (env) - argv [0];
...
snprintf (statusline,
statuslength,
"record: start recording %s to %s at %s",
channelname,
outfilename,
starttimebuf);
The results are visible in the ps output:
=== root@ceeveear (/dev/pts/1) /myth/Images 172 -> ps aww | grep record
15144 pts/0 S+ 0:00 record: start recording SBS HD to koeter-rex-2006-08-10
at 10 August 2006 19:30:00
| Thursday, 10 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More catch-up work today. What work shall I do now? I have a number of options, none of them overly attractive, but some still rather promising.
More investigation of the sound problems with HDTV, with some interesting non-results. I need to learn more about MPEG-2, but it's clear that multiple data streams are multiplexed in packets, and that each packet header contains a package ID or (confusingly) pid defining the stream to which it belongs. There seems to be no particular standardization of the pids for audio and video. mplayer thus investigates the beginning of the stream to determine the pids involved in the stream; somehow it also manages to determine their nature, by a means I haven't investigated yet. With the -v option, it shows what it chooses. For Channel 10 TV (normal definition), I get:
Checking for PVA Checking for MPEG-TS... TRIED UP TO POSITION 0, FOUND 47, packet_size= 188, SEEMS A TS? 1 GOOD CC: 32, BAD CC: 0 TS file format detected. DEMUX OPEN, AUDIO_ID: -1, VIDEO_ID: -1, SUBTITLE_ID: -1, Checking for MPEG-TS... TRIED UP TO POSITION 0, FOUND 47, packet_size= 188, SEEMS A TS? 1 GOOD CC: 32, BAD CC: 0 PROBING UP TO 2000000, PROG: 0 VIDEO MPEG2(pid=512)AUDIO MPA(pid=650) NO SUBS (yet)! PROGRAM N. 0By contrast, 10 HD gives:
VIDEO MPEG2(pid=514)NO AUDIO! NO SUBS (yet)! PROGRAM N. 0
So it didn't find the audio pid. But the video pid is different too; it's different on other channels as well:
| Channel | Video pid | Audio pid |
| 7 | 1281 | 1282 |
| 9 | 512 | 650 |
| ABC | 512 | 650 |
| ABC2 | 2307 | 2308 |
| SBS | 161 | 81 |
| SBS 2 | 162 | 83 |
| SBS HD | 102 | 103 |
Clearly it's difficult to guess the pid, especially since it can get quite big. But ~tzap/channels.conf contains lots of information. Could it include this stuff? Looking at it (and removing a lot of stuff in the middle), I found:
ABC HDTV ... :2314:0:592 ABC2 ... :2307:2308:594 ABC TV ... :512:650:595 7 Digital ... :1281:1282:1360 7 HD Digital ... :1345:0:1364 NINE Digital ... :512:650:1105 NINE HD ... :513:0:1112 TEN Digital ... :512:650:1617 TEN HD ... :514:0:1624 SBS HD ... :102:103:832 SBS DIGITAL 1 ... :161:81:833 SBS DIGITAL 2 ... :162:83:834 SBS RADIO 1 ... :0:201:846
Clearly the third last and second last columns are the video and audio pids. But what's the last column? It looks like a pid too, especially since exactly those channels which I can't get audio for don't have an audio pid, and SBS radio (the last line) has no video (not surprisingly) and the last two columns both set; in fact, all channels, including the ones I don't have set here, have the last column set.
So what is this? Some kind of digital encoding, MP3 maybe? The abbreviation MPA in the successful case may be a clue. The obvious thing to do is to read the tzap documentation. But, as I have complained in the past, that doesn't exist. OK, find the source. That's also more easily said than done, and by the evening I hadn't found anything up-to-date. apt-get said:
=== root@ceeveear (/dev/pts/1) ~ 265 -> apt-get source dvbutils
E: You must put some 'source' URIs in your sources.list
That will have to wait for another day.
| Friday, 11 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Up early this morning to go to Hahndorf and visit Peter Jones and Peter Denton, who have an interesting video recording project. Looks as if that might be interesting to work on. While I was there (just round the corner from Grumpys), took a bottle of my latest Weißbier for tasting; I was rather proud of it. Not for the first time, though, it seemed completely have lost its character on the short ride from Echunga to Verdun. I wonder why.
In the afternoon, more work investigating MPEG, but came to no conclusion. The two Peters came along a little later with some DVDs to evaluate. They have some corruption (manifested as pixellation during playback) that we need to investigate. That fits in very well with my current investigation of the MPEG format.
I must do something about my firewall setup; it's becoming increasingly clear that it's misconfigured. As if to rub salt into the wound, I was caught by a phisher who got my eBay password before I realized what was going on. Fortunately I recognized it pretty well immediately and was able to change the password immediately, almost certainly before any harm was done. But it brings home to me how easy it is to fall for one of these traps.
| Saturday, 12 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Spent some time this morning looking at my firewall. Still couldn't get pf to do what I wanted—I suspect that the problem is that that I'm only running a single Ethernet interface on my gateway machine, along with IP-IP encapsulation, so a lot of the assumptions fall down (for example, “data in” and “data out” on the xl0 interface are not the same, but the both happen for any packet that goes to another machine. I'll look at that later, but for the short term decided to revert to ipfw, which I stopped using about a year ago. I could have sworn that I ran an ipfw firewall after installing ADSL, but the configuration file I found clearly showed a PPP configuration. Spent far too much time fixing that, to discover that I'm currently the target of about one malpacket per second:
Aug 13 10:20:39 echunga kernel: ipfw: 11000 Unreach 13 TCP 222.145.63.155:3820 192.109.197.47:445 in via gif0 Aug 13 10:20:40 echunga kernel: ipfw: 11000 Unreach 13 TCP 219.150.243.140:39613 192.109.197.140:1433 in via gif0 Aug 13 10:20:40 echunga kernel: ipfw: 65000 Deny UDP 66.91.132.74:42359 192.109.197.255:38293 in via gif0 Aug 13 10:20:40 echunga last message repeated 5 times Aug 13 10:20:41 echunga kernel: ipfw: 11000 Unreach 13 TCP 222.133.10.66:53941 192.109.197.146:80 in via gif0 Aug 13 10:20:41 echunga kernel: ipfw: 11000 Unreach 13 TCP 72.11.40.243:56441 192.109.197.92:135 in via gif0 Aug 13 10:20:41 echunga kernel: ipfw: 11000 Unreach 13 TCP 72.11.40.243:56442 192.109.197.92:1025 in via gif0 Aug 13 10:20:42 echunga kernel: ipfw: 11000 Unreach 13 TCP 222.145.63.155:3820 192.109.197.47:445 in via gif0 Aug 13 10:20:43 echunga kernel: ipfw: 11000 Unreach 13 TCP 219.150.243.140:39613 192.109.197.140:1433 in via gif0
It looks as if it was mainly the otherwise good authentication that I run that has kept me out of harm's way.
In the afternoon, more investigation of the “no sound” problem on HDTV. Spent quite some time determining that yes, it seems that I did have the latest version of dvb-utils. Documentation is horrendous: even the name of the package is not clear. Debian calls it dvb-utils, but elsewhere it's called dvb-apps or dvbtools, and the package itself is called linuxtv-dvbtools. After a lot of searching, discovered the cause of my problems: the audio modulation for HDTV in Australia is different, but the information I need is not present at all in the channels.conf file: the last field is the service ID, whatever that may mean. Instead, HDTV uses (only) AC3 audio encoding (the other programmes also apparently do so), and dvbscan doesn't look for that when run in the “build channels.conf” mode. Instead, you need to tune to the stream with tzap and then run it in a different mode:
=== root@ceeveear (/dev/pts/0) /myth/Images 262 -> tzap -r 'ABC HDTV' -S &reading channels from file '/root/.tzap/channels.conf' [1] 23737=== root@ceeveear (/dev/pts/0) /myth/Images 263 -> dvbscan -cusing '/dev/dvb/adapter0/frontend0' and '/dev/dvb/adapter0/demux0' 0x0000 0x0250: pmt_pid 0x0102 ABC -- ABC HDTV (running) 0x0000 0x0251: pmt_pid 0x0100 ABC -- ABC TV Adelaide (running) 0x0000 0x0252: pmt_pid 0x0101 ABC -- ABC2 (running) 0x0000 0x0253: pmt_pid 0x0103 ABC -- ABC TV (running) 0x0000 0x0256: pmt_pid 0x0105 ABC -- ABC DiG Radio (running) 0x0000 0x0257: pmt_pid 0x0106 ABC -- ABC DiG Jazz (running) dumping lists (6 services) ABC HDTV (0x0250) 01: PCR 0x0905 V 0x090a TT 0x090c AC3 0x090b ABC TV Adelaide (0x0251) 01: PCR 0x0080 V 0x0200 A 0x028a (eng) TT 0x0240 AC3 0x0294 ABC2 (0x0252) 01: PCR 0x0902 V 0x0903 A 0x0904 (eng) TT 0x0906 ABC TV (0x0253) 01: PCR 0x0080 V 0x0200 A 0x028a (eng) TT 0x0240 ABC DiG Radio (0x0256) 02: PCR 0x0901 A 0x090d (eng) ABC DiG Jazz (0x0257) 02: PCR 0x0901 A 0x090e (eng)
Conveniently, this output is also in hex; presumably somebody else wrote it. Putting the AC3 PID into channels.conf finally gives audio. But what a pain to find that out!
| Sunday, 13 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More work on the stuff I learnt yesterday, mainly updating the LinuxTV wiki with pages for dvbscan and tzap. In the process discovered that we're not out of the woods yet; sometimes mplayer still doesn't find the audio in HDTV streams.
Playing HDTV also brought me up against another issue: it looks like my machine isn't fast enough, and I get lots of dropouts:
a52: CRC check failed! -0.354 ct: 0.002 4/ 4 ??% ??% ??,?% 0 0 a52: error at resampling a52: CRC check failed! -0.108 ct: -0.050 30/ 30 59% 27% 6.5% 3 0 a52: error at resampling a52: CRC check failed! -0.147 ct: -0.162 89/ 89 49% 13% 5.1% 12 0 a52: error at resampling
Tried running it on eucla under Fedora Core 5, where it didn't have any problems with dropouts—but exactly every ten seconds it hung for about 5 seconds, and then continued as if nothing had happened. I suspect that's some artefact of GNOME, so rebooted the machine into FreeBSD to try it there—but of course mplayer wasn't installed. Spent some time trying to do that, but didn't finish.
| Monday, 14 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Finally got mplayer installed on the FreeBSD incarnation of eucla and confirmed that it didn't have any difficulties with the HDTV MPEG stream. So the simple answer seems to be: throw more CPU at the problem. eucla has a 2.0 GHz Pentium M; teevee has a (1.466 GHz) Athlon XP 1700+.
Getting through mail is becoming easier. For the first time in months, I got down to less than a single page of messages index (82 messages). Also various preparations for the rest of my life.
Due to a comedy of errors, the air conditioner fan that failed last year and which I picked up a month later, has still not been repaired. Finally got the insurers to send out Alan Larsen, the bloke who looked at it last year. He claimed it had burnt out and took it off back to the rewinders.
Yana back in the evening for a brief visit.
| Tuesday, 15 August 2006 | Echunga | Images for 15 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Into Hahndorf again this morning to talk with Peters Jones and Denton about our project; came up with a project plan which looks like it's achievable, but it took us 4 hours.
Alan from the insurers back again today with the fan. It now worked, but turned in the wrong direction. Alan quickly trimmed the cable with materials at hand, making a horrible mess of the cable in the process, and got it to turn in the right direction—but the cable was 50 cm too short. Took it away again.
Why am I so tired?
| Wednesday, 16 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Spent an inordinate amount of time today setting up a ssh tunnel to the test machine in Hahndorf. The machine is behind NAT, so I can't contact it from here; instead it needs to set up the tunnel from its end and let me log in. That's fine, except that we want it to be automatic. Solved the problem like this:
ssh -fN -R 4096:&myhostname>:22 <remotehostname
su - tunnel /home/tunnel/tunnelout
The real issue is that this expects port 4096 on the remote host to be available. If it's not, it won't work. I haven't thought of a good way to work around this. I could use trial and error, but I don't see that as a good way.
Alan back again with the air conditioner fan and a new cable, and mounted it with some difficulty. It works! Not quite 15 months since it failed.
| Thursday, 17 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
One of the less pleasant aspects of my new work is that the other two members of the team are Microsoft users, and I frequently get documents from them created by Microsoft products. I asked for PDF instead of Microsoft “Word” and “Excel”, and got it—but what printed out was completely illegible, probably a font problem. In any case, updating a single copy of an “Excel” spreadsheet is not practical, so decided to convert it to a (single) MySQL table and access it via the web with phpMyEdit. Peter Jones came up with the alternative of DataFace (why do all these product names have strange capitalization?), which at least has nice-looking documentation, but I decided to start with phpMyEdit because I had used it successfully last month, and then compare DataFace.
The first step was to convert the CSV-format “Excel” output to MySQL IMPORT statements; that proved to be less easy than I had hoped, due to differences in the way “Excel” treats things. Tried a (Microsoft-based) program called, well, Excel MySQL Converter—one of the most broken web sites I've seen. I'd guess that they use fonts only available on Microsoft machines, where it displays OK. It offered free download, so I tried that. Yes, the download is free, but it's a demo version that only outputs (correctly in my case) 5 rows. The real software is $40, something I don't like paying. Ended up cobbling the thing together with Emacs and finally had a table in the correct format.
By comparison, setting up phpMyEdit was a piece of cake. It seems that you really need all the data in the directory where the page is, because it refers to it, but it's happy with symlinks. Setup to create the web page only took a couple of minutes, and the result looks like just want we want, modulo some strangeness with the number of lines per page, which varied from one to the next. Still, such a success that I decided to defer examining DataFace.
| Friday, 18 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More spam from Mindspring. As usual, I bounced the message, with all headers, to abuse@mindspring.com. I got a message back within minutes:
Thank you for submitting a report to the EarthLink Network Abuse Department. Unfortunately, your submission does not contain sufficient information to determine the nature of your issue. Evidence to Abuse should always include at least the IP address of the offending party and a valid timestamp, which includes time, date and timezone.
Idiots! That's what I replied to this message. I got another automated reply, of course, but not quite what I expected:
Thank you for submitting a report to the EarthLink Network Abuse Department. Your ticket number is AB0000000650839, which you'll need to reference if you email us again regarding this issue.Sounds like they recognize this one.
Playing around with the cxm driver for Hauppauge PVR-350: we found that if you pipe it directly into mplayer, it suffered severe buffer overflows, giving really bad images. Modifying the number of buffers in the ring significantly improved that; we'll have to look at what's really needed.
Also looked at the mplayer port, about which I have said less than friendly things in the past. I have only just discovered that the version in the Ports Collection is version 0.99.8.2, which is so old that I can't even find a date for it; but 1.0pre7 came out well over a year ago, and 1.0pre8 came out recently. Did some attempts to build that, with the usual problems of port dependencies getting in my way. Put it on quartet.lemis.com, a test machine with (almost) no ports on it, and left it go (it's a quad CPU 200MHz Pentium Pro).
Over to Hahndorf to look at the machine, which has a combination of ATA and SATA drives, along with four SATA ports and a single ATA controller. Lots of fun configuring the BIOS, and what we ended up with in the end is not necessarily the ideal.
More playing around with phpMyEdit in the evening, and imported our deep freeze contents list, previously stored in a flat file, into MySQL and set up a web page to maintain it. Again, it went very quickly, though on second view it's a little clunky to update things. Maybe I should take another look at DataFace after all.
| Saturday, 19 August 2006 | Echunga | Images for 19 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Writing up my diary entry for yesterday gave cause to investigate the versions of mplayer I was running. Looking at the build log file, I found:
install -o root -g wheel -m 555 /src/FreeBSD/ports/multimedia/mplayer/work/MPlayer-1.0pre8/
TOOLS/wma2ogg.pl /usr/local/share/mplayer/tools
install -o root -g wheel -m 555 /src/FreeBSD/ports/multimedia/mplayer/work/MPlayer-1.0pre8/
TOOLS/x2mpsub.sh /usr/local/share/mplayer/tools
...
===> Compressing manual pages for mplayer-0.99.8_2
...
===> Registering installation for mplayer-0.99.8_2
So it is the latest version, only the reported version number is wrong!
Following up on my HDTV sound problems of a few days ago, discovered that a programme I had recorded from 9 HDTV still claimed to not have any sound. I'd seen intermittent problems in this area, so kept the program for investigation. As it turned out, it worked fine on wantadilla, but not on teevee, which is running 5.3-RELEASE. It also has problems with a bad sector on the main disk, so it's a clear candidate for an upgrade. I don't have a spare machine, and I don't want to take teevee down for an extended period, so chose ceeveear instead—the only thing that it's doing is recording the occasional programme, and I can always reboot it for that. Chose a new disk and installed 6.1-RELEASE, then upgraded to 6.1-STABLE. In the process it came home to me just how little I have achieved in the last two years in my attempts to create a good system upgrade procedure.
Yvonne into town today and finally found the correct connectors for the carbon dioxide cylinder:
Connected it up with the regulator, turned it on and—the gas shot out of the pressure relief valve. I wonder what's gone wrong there.
Kirk McDonald sent me a couple of hop rhizomes a month or so ago: one (East Kent?) Golding, one Hersbrucker (as Kirk said, “No, Hersbrucker, not Hallertau”).
Hersbrucker are one of the best-known hops from the Hallertau region.
Spring's early this year, but it would have been time to plant the new hops anyway, so did that in the afternoon.
| Sunday, 20 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Planned a quiet day today, but somehow it didn't quite happen that way. Instead, continued with the rebuild of teevee, which proceeded slowly but steadily and took up most of the day. In the end had swapped the hardware of ceeveear and teevee. As a result, I can now watch HDTV with audio and only use about 50% of the CPU. In combination A number of things have probably contributed to this:
The CPU is more powerful (Athlon XP2500+ instead of XP1700+).
I'm using the latest version of mplayer, which appears to perform many fewer system calls, only a few thousand per second; my recollection with the previous version was that there were up to 100,000 calls per second, and that with normal defnition TV.
I'm now using the nVidia GeForce 6200 card that I bought a couple of months ago, but which I couldn't get the Linux software to handle correctly.
Also managed to get rid of the annoying noise at the bottom of the display: turned out to be a mode line issue: I didn't wait long enough at the bottom of the display. With the help of xvidtune got rid of it completely, so I'm now running 1280x720 as intended, and not the 1280x736 that I had been running for over a year. Here's the mode line for my Panasonic PT-AE700E:
ModeLine "1280x720" 99 1280 1324 1420 1656 720 725 728 772 +hsync +vsync
| Monday, 21 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
MySQL may be gone, but Mondays aren't; still, it's interesting to see how much less work there is to do, notably with mail. My overall mail statistics haven't changed much, but the number of messages I need to attend to in detail has.
Set up named on the Hahndorf machine; like most Microsoft oriented LANs, the DNS was a mess—no reverse lookup, for example. Setting up named is not as easy as it used to be, either. A lot of things have changed, and last time I looked there wasn't much in the way of documentation. I should take a look again, though: the information in the book is obviously out of date.
Also looking at the CVS repository, paradoxically maintained on a Microsoft machine. Some strange things have happened there, and I probably need to start a lot of stuff again.
Some discussion on the home brewing lists about what was wrong with the regulator on my gas cylinder; some people claimed that the dip tube was still in the cylinder, and that this was—by some mysterious means—causing the problems. Considering that only gas came out of the cylinder, it's a moot point whether the dip tube is there or not. Finally found it: I just hadn't tightened up the connection enough. The gas wasn't escaping via the overpressure valve at all, just from the connector to the bottle, and in such a way that it appeared to be coming from the valve. Clearly these connectors aren't intended to be left merely hand-tight. Egg on Groggy's face.
| Tuesday, 22 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Spent a lot of time today setting up a CVS server, something I've never even looked at before. A quick Google brought up Dan Langille's HOWTO, which pointed at a more detailed book on the subject, Open Source Development with CVS. That's probably a little more detail than is needed. The main problem is that there's no tool to create the CVSROOT/passwd file. The links recommend a perl script, but I'm not into perl, and it seems a bit over the top anyway.
The setup's pretty straightforward:
Enable the entry for CVS pserver in /etc/inet.d. Specify a path to a repo or series of repos (the actual repo is specified by cvs login; the path is just to restrict access to that hierarchy). In this example I've used repos in /src/cvs:
cvspserver stream tcp nowait root /usr/bin/cvs cvs --allow-root=/src/cvs pserver
HUP inetd to enable the entry.
Create a CVS repo under /src/cvs.
Add a user cvs to the system password file /etc/master.passwd (using adduser or whatever).
Using passwd, give cvs a password intended for user foo.
Create an entry in CVSROOT/password and copy in the password for cvs from /etc/master.passwd:
foo:$1$0fNgUo0H$cqxxmfFLGYJKttv1gwO5b1:cvs
This enables foo to connect using the password you've assigned, and to run the CVS checkout operation as cvs.
Repeat the password change for other users, then change it again to something that none of the users know.
| Wednesday, 23 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More work with the CVS repository today, which kept me busy. Then on to look at the build structure. It's still in the kind of condition where it's easy to make quick progress, and by the evening things were looking a lot better.
| Thursday, 24 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More work on CVS today, including some strange authentication problems that I didn't get solved.
Yvonne into town this afternoon to pick up some pieces for a new, real CVR, based on a “bare bones” system from MSI. Amusingly, this relatively low-end machine is my first own AMD64 machine (I had a dual Opteron machine last year, but it belonged to Rocksoft). The current box looks interesting; it's almost the size to fit into a home entertainment tower, though I fear it might be a little deep. Hopefully it won't be too noisy.
| Friday, 25 August 2006 | Echunga | Images for 25 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Spent most of the day today installing the new CVR machine. Its internal layout is quite interesting: there's a removable cage at the front for disk and DVD drive (and, puzzlingly, for a floppy disk, made in such a way that you can't use it for a second disk). The remainder looks quite well laid out too:
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Putting the machine together wasn't without its surprises. First, I ended up with the wrong memory (DDR-2 instead of DDR, so I had to take some memory out of teevee), and secondly I found two fans, one that came with the CPU and one that came with the system:
|
|||||||||||||||||||||||
The (larger) one on the left is just sitting on the PCI slots for comparison: that's the one that came with the CPU, but it doesn't fit properly on the CPU (too loose). I wonder how many people have lost their CPU as a result, and why MSI made such an incompatible board.
Installing FreeBSD on the machine brought a couple of surprises, too: read errors from the installation DVD, and later a panic with corrupted data. Guessed at memory incompatibilities, but the SIMM I put in was a PC 400 SIMM, and that's what the BIOS was set for. Turned the memory speed down to 166 MHz and had no further stability problems.
Setting up X was the next issue: the board has its own display chip, but x.org doesn't recognize it. It managed to set it up with a really unusual resolution, something like 1800x1350, and mplayer wasn't able to render correctly on it. Put in an SiS 315 board, set to 1280x720, and it worked.
But, not for the first time, it seems that 64 bit machines are slow! This is an Athlon 64 2800+, presumably intended to be faster than the Athlon (32) 2500+ in teevee; but tests with a specific HDTV file showed much more CPU usage. Of course, there are a number of things to compare: display card, display drivers, memory timing and 64 bit mode. I'll do them in sequence.
| Saturday, 26 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
I had intended to go riding today, but it was cool, and somehow I managed to find excuses not to do so. Instead carried on looking at the new CVR machine, and did numerous tests with mplayer, making interesting discoveries on the way:
The nVidia driver does not improve performance; in fact, it decreases it.
The nVidia 4000 card does not perform any better than the SiS 315; if anything, it's slightly worse.
Adding memory does not improve the performance. This was unexpected: I hadn't expected the system to require more than the 512 MB that I had, but with two memory SIMMs, the chip set can set an interleaved mode that should give better throughput.
Increasing the memory bus frequency from 166 MHz to 200 MHz (which it should have been) also didn't make a perceptible difference.
Removing the sound output (-nosound option) also didn't make any difference. That's not surprising, but I tested it because mplayer, in its obnoxious “ **** Your system is too SLOW to play this! ****” message, claims that it can make a difference:
- Most common: broken/buggy _audio_ driver
In summary, nothing I did significantly improved the situation. The next step was to install a 64 bit kernel; but that failed miserably. The instability I had noticed before became impossibly bad with the 64 bit kernel. Both FreeBSD and Ubuntu didn't make it through the installation process. Spent a lot of time playing with BIOS settings (the “restore default settings” and “set optimized defaults” both seemed to do nothing), and finally removed all boards, used the new disk I had bought for the machine (previous attempts were done with disks about 3 to 5 years old, but still in good condition) and tried again—and it worked! So: is this a problem with the BIOS settings, some of which I might have reset after all, with the boards, or with the disk? Who knows? Still more work to do.
| Sunday, 27 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Brew day today, but also found some time to work on the CVR machine. Spent a lot of time working on the system upgrade scripts, but also installed X—strangely, it created a configuration file very different from the one it created on the same hardware in 32 bit mode. Did some preliminary tests with mplayer suggested that it was a lot faster than in 32 bit mode.
But before I could start some real tests, it crashed. Not once, but repeatedly; it took several attempts before it would stay up long enough to finish the boot and fsck. And the only way I could get it to stay up longer than about 5 minutes was to turn all the timings down to their minimum values, which also slowed the machine down unacceptably. Finally, frustrated, gave up. What a waste of time!
Sad news in the evening: Yvonne's father, Henner Ködderitzsch, is dead. He was 85 years old and in very poor health, so in many ways it was the best for him, but it's still sad.
| Monday, 28 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
What a waste of a day! Spent most of it double and triple checking what's wrong with the new machine, without any firm result except to confirm that it's not working correctly. That's a particular nuisance because it's holding me up doing other things, notably video performance testing.
| Tuesday, 29 August 2006 | Echunga | Images for 29 August 2006 |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Into town with my defective computer today and spent quite some time at the warranty/service department, where I was first told that the processor that I had been given was a 32 bit Athlon—this after I had said that it had run for some time in 64 bit mode—and then that it was not the chip that I had ordered. In the end it eventuated that the motherboard BIOS didn't know about the processor—which of course was an Athlon 64, as the photos show—and thus couldn't set the operating parameters correctly. It seems that Socket 754 chips are pretty much unobtainable nowadays, so in the end we agreed to return the machine and I started again with the same cabinet, an MSI Socket AM2 motherboard and a faster processor (Athlon 64 3500+).
Then to AFA to talk with Tony Arthur and Kym Swanson, and home to put the new machine together. That wasn't without its surprises either. It worked fine in 64 bit mode, but X didn't load the driver for the on-board video chip (part of the VIA chip set), and the sound driver didn't detect anything. Some research showed that the via driver isn't 64 bit clean. Damn.
Putting in an AGP card wasn't an option, either: this board has PCI Express slots, two of them, but I don't have any PCI Express boards. Finally removed a PCI card from wantadilla and used that. Result: I can now finally display the test HDTV image without dropping frames. Now to investigate the other issues.
| Wednesday, 30 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
Spent most of the day playing around with the new motherboard. Far too much of it isn't supported. Booted back into 32 bit mode and found that I still couldn't load the via driver: it seems that this chip set is too new. I was also unable to find a FreeBSD driver that would recognize the sound chip, which seems to be intimately related to the VGA chip. pciconf -lv reports:
none1@pci1:0:0: class=0x030000 card=0x72531462 chip=0x32301106 rev=0x11 hdr=0x00
vendor = 'VIA Technologies Inc'
class = display
subclass = VGA
none2@pci4:1:0: class=0x040300 card=0x72531462 chip=0x32881106 rev=0x10 hdr=0x00
vendor = 'VIA Technologies Inc'
class = multimedia
Did get some testing done, which showed that the difference in mplayer performance between 32 bit and 64 bit versions of X was less than the margin of experimental error: so for the moment it doesn't seem worthwhile taking in the disadvantages of 64 bit mode (mainly less driver support).
After that, decided to install Linux, the latest version of Ubuntu (“Dapper Drake”). They've managed to dumb down the installation to a point where it's not possible to set anything except possibly the partition size. How I hate these “modern” GUIs!
To keep me from boredom, got a message from somebody asking me if I could scan something in for him. The answer proved to be “no”: the disk in pain.lemis.com, the Microsoft laptop that I use for scanning and reading Microsoft format mail, had died. Dragged some stuff together and managed to halfway reinstall, but I'm missing some disks. Looks like I'll have to start again.
| Thursday, 31 August 2006 | Echunga | |
| Top of page | ||
| previous day | ||
| next day | ||
| last day |
More stress today; fortunately the planned meeting of the ICT Council for South Australia was cancelled.
One of my daily cron jobs shows various historical information. Today one was significant:
Third of a century ago: Tuesday, 1 May 1973 May 1 Greg starts working with UNIVAC, 1973
That was my first job. It's been a while; interestingly, half of it with some form of UNIX on my desk top. It seems that others are also thinking of the past; Charles Hannum of the NetBSD project sent a long message to the BSD projects painting a bleak picture of the future of NetBSD. It's too long to reproduce here, but some of his points are quite valid. It's been 15 years since Linus Torvalds announced what became Linux (it seems like only yesterday that it was 10 years). It's also been nearly 15 years since I first installed BSD/386 at home and quickly moved to BSD as my main operating system. In that time, the whole concept of computers on desktops has changed, and the idea of people writing software for free has dropped in popularity, especially since rival “Open Source” projects manage to pay a majority of the contributors. I wonder where we'll be in another five years.
As the result of a suggestion by Stephen Rothwell, went looking on the VIA web site and found sources for VGA drivers for the chip set I'm using, released under what is effectively a BSD license. That's in marked contrast to the claims on the web that VIA hasn't understood “Open Source” and provide no support for them. I wonder why the drivers aren't included in the x.org distribution, and whether they're in the XFree86 distribution.
Tried installing the latest version of KnoppMyth. The installation process is no more refined than the previous version which I tried a couple of months ago . The “automatic” install fails with:
/usr/local/bin/KnoppMyth-auto: line 609: / 10000000: syntax error: operand expected (error token is "/ 1000000").
Looking at the code, it's in partition_autosize() trying to calculate the disk size based on a variable $SIZE that obviously didn't get set. It's set by the output of fdisk, and it seems to assume that the disk in question is /dev/hda; in this case, for some reason probably related to SATA, it's /dev/hdc. But I want to install, not debug installation procedures.
Tried the custom install and on leaving the Partition-Menu (sic), got
Fatal dialog error: Error: Expected at least 7 tokens for --radiolist, have 4.
I should be used to this by now, but somehow I still find it incredibly frustrating.
Do you have a comment about something I have written? This is a diary, not a “blog”, and there is deliberately no provision for directly adding comments. It's also not a vehicle for third-party content. But I welcome feedback and try to reply to all messages I receive. See the diary overview for more details. If you do send me a message relating to something I have written, please indicate whether you'd prefer me not to mention your name. Otherwise I'll assume that it's OK to do so.
| Top of page | Previous month | Greg's home page | Today's diary entry | Next month | Greg's photos | Copyright information |