New system installed

The long-planned and long-prepared new system has been finally installed to replace the aging old hardware which had often observed unexpected recording terminations after 5-years of its continuous operation.

The hardware specification is found in the previous page. Initially, the new system still used old Hauppauge analog TV cards to record analog cable signals that were converted from digital by the cable TV company. The old software was used without changes.

Remarkably, the new system shows far superior performance numbers to the old one.

• Ffmpeg transcoding is 6x faster
• File transfer over the Internet is 4 times faster (look at the speed chart on right). 

Then, about 4 months after the new hardware installation, I finally succeeded in capturing and view digital TV broadcast all in digital.

BREAKING NEWS

Real-time streaming of live TV programs broadcast in Japan is now available for both PC and TV (with a Google TV STB).

I will follow this up in the Japanese version of this blog.

Stay tuned.

Digital Software

Operating system

I chose the Fedora 14 distribution (FC14) of Linux because I have been using RedHat distributions of Linux for a long time since the RedHat 5, if my memory still serves, and I have several in-house system administration tools in my inventory (my first serious Linux experience was Slackware). These days  many software developers prefer Ubuntu. In fact, I have been using Ubuntu desktop editions at my workplaces for the last four years. Either distribution has its own pros and cons, but my personal impression is that the Fedora is better for server type applications while Ubuntu is good for desktop and workstation applications. Fedora's System V style RC scripts are very orthogonal and each subsystem is well-defined. This makes system administration very clear while Ubuntu sets up many shortcuts for casual use. For example, on Fedora, "service network start" is merely an alias for "/etc/rc.d/init.d/network start" and it only sets up network interfaces. On the other hand, Ubuntu's "service networking start" and "/etc/rc.d/networking start" work differently and the latter even starts the NTP daemon. This is why my three home servers, central, firewall, and backup, run Fedora, but I installed Ubuntu to my laptop using VMWare.

One argument was which version of Fedora, FC13 which has been installed on all of my servers, or FC14, the latest as of the time of my development. It would make my life a little bit easier if I had chosen FC13 because of unity, but I could not resist the idea of "the newer, the better." The TV recording server will not have an OS upgrade once it is installed during its hardware life time, so its OS version will diverge from other installations anyway.

I must confess that FC14 was a bad choice and I suffered a long delay from it buggy PC card reader package. It cost me nearly 2 months and $60 to figure out the defective PC card reader package and to start using the new digital recording system. If you are to build a new system avoid FC14, use other versions.

Another caveat with the Fedora distro is its short support spans. The fedora project mentions its goal is to introduce as much up-to-date stuff in a timely manner as possible and to release new versions every 6 months and an older version will lose update support in 1 year or after it successor is released. This is a bit too short for a practical server applications like mine. I heard CentOS is very compatible with Fedora with much longer support spans, so I may choose CentOS next time.

Software modules

The digital recording software roughly consists of the following components:

1. Recording scheduling front-end (GUI)
2. PT2 tuner card device driver
3. TV recording executive
4. ARIB25 decoder
5. TS (transport stream) splitter
6. Transcoding engine
7. File transfer
8. Recoded program file management
9. Media server

Development strategy

I have been developing and maintaining my home IT systems for years. This TV recording system is one of those and a private (i.e. non-commercial) project. Probably like many other home systems, I was not paying much attention to maintainability of my development. Most changes were made on ad-hoc basis, and I only used RCS (revision control system) in each individual directory for maintaining file change history. Recently, I have changed my strategy and started using more systematic approach for tracking my development because my home system is now distributed to three servers after upgrading both hardware and software platforms.

VMware

Fedora 14 on VMware – How to Install VMware Tools on Fedora 14

Using Shared Folders

Note: You can use shared folders to share any type of file. However, Windows shortcuts and Linux symbolic links do not work correctly if you try to use them via shared folders.
Caution: Do not open a file in a shared folder from more than one application at a time. For example, you should not open the same file using an application on the host operating system and another application in the guest operating system. In some circumstances, doing so could cause data corruption in the file.

DIgital System Hardware

The design goals of my new digital TV recording system are:

• Receiving, recording, and transferring Japanese digital terrestrial TV broadcast to my home server in all-digital means
• BS/CS satellite broadcast reception is optional but not a primary goal
• Use of the existing or improved point-n-click GUIs for TV program scheduling
• Use of the existing infrastructure for transferring and managing recorded TV program files
• 10 years or longer system life time with possible hardware and software maintenance and replacements
• At least 5 years of hardware life time

The current analog system has been doing an extraordinarily remarkable job; it's based on an advertisement-special CPU/mother board combination and a non-RAID disk, but it's been working continuously on a 24/7 basis for more than 5 years except one hardware downtime due to a faulty power supply. The new system should be better than this.

Earthsoft PT2

The system hardware consists of the following main components:

• A modern x86-based Linux server box
• Earthsoft PT2 digital broadcast receiver PCI card (1 ea.. One card can receive two terrestrial and two satellite signals simultaneously)
• B-CAS IC card (1ea.)
• NTT-ME SCR3310-NTTCom IC card reader (USB)

I purchased both the Earthsoft PT2 and NTT-ME IC card reader from Amazon Japan. You cannot purchase a B-CAS card by itself; it always comes with a B-CAS-certified commercial product. I happen to have purchased a Pixela PRODIA digital tuner so I decided to use the card that came with the PRODIA.

NTT-ME SCR3310-NTTCom

Here's the server's specification:

CPU:	Intel Core i5-650 3.2GHz (2 cores, 4 threads)
Motherboard:	Intel DH55HC (3 PCI32 slots)
Main memory:	2GB
Digital capture card:	Earthsoft PT2
Analog capture card:	Happauge PVR350 x2
Disk drives:	WD 2TB x3 RAID1 (usable capacity: 2TB)
Total cost:	Around $1300US

CPU

The choice of CPU is mostly a personal matter, as long as it can provide enough horsepower. My old analog system uses an AMD Sempron 3000+ (1.8GHz) single-core CPU that spends about 2.2x time to transcode an MPE2 stream to an MPEG4 AVI file (a 10-minute program needs 22 minutes for transcoding). A rough benchmark test of the new system showed only 20% of the program time, i.e. more than 10 times of speed improvement. A less powerful CPU could be still appropriate, but the new system not only is supposed to perform ARIB25 decryption, TS segment extraction, and transcoding for each recorded TV program, but also is planned to be used as a remote backup server that requires fairly CPU-intensive tasks of rsync-over-ssh on a VPN to write into an encrypted file system.

On the other hand, higher-end Intel CPUs such as the Core i7 do not have on-chip GPU, and only expensive high-end GPU cards are available on the market these day as more and more low-end CPUs include an on-chip GPU. A recording server needs no high performance graphics and I wanted save money. The CPU of my choice is the highest performance model with an on-chip GPU as of the purchase. I installed a Cooler Master dual-fan CPU cooler for the maximum cooling effect and reliability (fan redundancy).

Memory
My current analog system has 512MB, out of which 64MB is dedicated to video memory, and runs with a moderate swap amount (about 150MB). Even though the new system is to allow simultaneous multiple transcoding sessions, 1GB will be probably enough. But it's always true that the more the better, and, unlike a local system to which I could add more memory later when I find memory shortage, the system will not be upgradable. So, I gave 2GB,

I have reserved 1.5GB (512MB x3 striped on 3 disks) for the swap space. This may look too small for a system with 2GB of main memory. From my experience, however, if a system actually uses more than 1GB of a swap space, the system has entered a thrashing state (the whole system is spending majority of its processing power for swapping data between the main memory and the swap space) and the system is virtually unusable. Adding more memory is far better.

Motherboard

These days, the majority of motherboards available on the market are mATX-sized with only one or two PCI32 slots. But I needed at least three PCI32 slots to house one PT2 digital TV card and two PVR350 analog TV cards. Another considering point was compatibility with Linux. I have experienced driver incompatibility  with the on-board Ethernet chip on the current analog system's motherboard and on another Core i3 server system I had built a few months before (Gigabit  motherboard), both costing a PCI slot for an add-on Ethernet card. The Intel motherboard was indeed a good choice in all aspects.

Hauppauge Analog TV Cards

Hauppauge PVR-350

I expect that I will need to spend as long as a few months after installation for debugging until the new digital recording system becomes stable enough. The new system must should be usable providing the same service of analog recording during this transitional period. So, I included two PVR-350 cards, the same ones as the current analog system has. The PVR-350 includes unnecessary FM radio tuner and the less-featured (and less-priced) PVR-150 would be satisfactory, but it depends on the market prices on eBay since both are discontinued products.

Disk Drives
A disk drive is one of the most vulnerable components in a computer system as it has mechanically moving parts. My current analog system has had no disk failures for more than 5 years of continuous 24/7 operation, but I would say it's just a miracle. My other systems including the central home server suffered fatal disk failures. In order to keep the system up and repairable even at a disk failure, I treated the system with good redundancy using a three-disk RAID1. The usable capacity of 2TB is far more than necessary for a TV recording server that would need around a few 100s of GB, but I spared more for remote backup server use.

When building a RAID system, I would recommend not using the full disk capacity for RAID use but reserving a small space because the manufacturer may produce disks of slightly different disk size (number of bytes) under the same model name (but likely under different part numbers). When one of the RAID disks goes south and needs to be replaced, you will buy a disk with the same model name as the faulty one as long as it's still available on the market. If the new one has a slightly smaller capacity and you have allocated the whole disk space for RAID on the old ones, you have a problem. I learned this from my own experience on our central home server. So, I always reserve about 5% of the total disk space for absorbing this. It's a waste, but the whole purpose of RAID is to prepare for a rainy day with redundancy, isn't it?

Case
The computer case is much more of a personal choice. I chose the Cooler Master Elite 370, which I found with a reasonable price tag ($31US)  at a local store, and for the following reasons:

Cooler Master Elite 370

• Low profile (17"H, the lowest for a full ATX MB, good for the housing service's limited rack space)
• Good air flow

To extend the life of CPU components, especially disk drives, cooling is very important. When I started my first job as an R&D engineer at H.P. designing logic boards for smart instruments, I learned every 6 degrees of temperature increment shortens electronic components' life by half.  Since my system is going to be installed in a housing service's server room, fan noise is not a problem. So, I installed a total of 5 powerful (i.e. noisy) case fans with careful considerations about natural air flows. I don't know whether a bottom-mounted power supply, which is a new trend, is better than a traditional top-mounted PSU.

Power Supply
Some fanatic PC-builders seek more and more powerful power supplies, say 800W, these days. My 4-disk central server consumes about 80W on its UPS reading. So, any PSU rated 250W or more would be more than satisfactory. One note is to choose direct SATA power connectors rather than Molex-to-SATA adapters, not only for saving some extra bucks but for avoiding causes of hardware faults, which I have suffered before. A PSU will ware anyway. My old analog system had to replace a dead one, and so did most of my other systems.

Cost
So far, the total BOM has run about $1300US, about twice as the analog system. The PT2 digital tuner (24,800JPY or about $300US as of writing) carries the biggest bill. Other contributers are the CPU+M/B+fan (increased from a budget $100US kit to about $300US) and the RAID disks (from a single disk).

Proceed to New system installed.

Digital System

The biggest issue in recording Japanese digital terrestrial (as well as BS and CS, i.e. satellite) TV broadcast is its tight DRM (digital right management). Although broadcast reception itself is free (except NHK, the public broadcast service that collects monthly subscription fees but not on "pay-per-view" or other means), the digital contents are encrypted in a very tight way called "ARIB25." A single private company, B-CAS Systems Inc., controls the whole DRM, and it strictly prohibits decrypted clear test contents from passing general computer buses such as PCI and USB. Several major brands license from B-CAS and manufacture digital TV receiver add-ons, but the recorded content files are still encrypted with the individual PC-unique key so that the recorded contents can be viewed only on the PC that recorded them. It is not necessary to mention that all of these  officially licensed systems are for Windows only. This is a huge issue for systems like mine that are based on Linux and aim at freely transferring recorded contents.

There are a few possible solutions for recording digital broadcast:

1. Receive and record analog signals converted by the cable TV feed with the existing analog system
2. Set up a local digital-to-analog STB (set-top box) and use the existing analog system
3. Develop a brand-new digital system

In order to help those who cannot immediately afford a new digital-compatible TV and to save still-usable analog TVs from being dumped and trashed, the government asked cable TV companies to continue to broadcast analog signals by locally converting digital contents to analog. Most major cable TV companies have promised to do so for 3 to 5 years after cease of aired analog TV signals. This saves and extends my system's life for 3 to 5 years without any changes or additional investments, but only 3 to 5 years. After the grace period ends, no analog signals at all.

Pixela PRODIA PRD-BT102-PA1, a sub-5,000 yen digital TV tuner STB distributed by AEON

Using a local STB is supposed to require only relatively minor modifications in TV-station selection to my current analog system, so it should have a relatively low technical barrier. One of the issues was the cost of such an STB, which was about 20,000 yen ($250US as of writing) or more per reception (my system allows two simultaneous receptions). The government encouraged the industry to produce sub-5,000 yen (below $60US as of writing) STBs for free distribution to households with financial hardships and for those who wanted to continue to use their existing analog TV sets with minimum investment. Though the industry was initially skeptical about such products because of their high manufacturing cost (est. about 4,000 yen), at least three major businesses responded with their minimally-featured STBs. At one point in mid-2010, I almost decided to go this way as it appeared most feasible and practical, and I purchased two such STBs （Pixela PRODIA PRD-BT102-PA1, distributed by the AEON supermarket chain)　when I went to Japan for a family vacation. Other sub-5,000 yen tuners include the Buffalo DTV-S110 and Maspro DT630.

I, however, was reluctant to start a project to write additional software to control the STBs using IR remote sticks. Why? In addition to the technical difficulty in detecting whether the target STB's poewr is turned on or off (I never understand why such an STB needs to be turned off), writing software to control an STB with an IR emitter is one of the least exciting jobs. And the efforts and investment would reward nothing regarding to picture quality improvement except it would become ghost-less. Who would get excited to build a system to scan wallet-size pictures taken by a good digital SLR?

The all-digital solution is ideal only if the DRM issue is resolved. Quite a few talented people challenged this, and a few actually succeeded. One famous solution is Friio, a USB-base STB from a Taiwan-based business that accepts a B-CAS card and records digital broadcast in an unprotected and freely-transferable way. Others include a sort of radical hardware hacking called ”TS-Nuki"　（TS抜き） or transport stream extraction. Both are supposed to work only with Windows, but not with Linux.

I knew that one person finally cracked the ARIB25 encryption system and published his work, i.e. Linux source code, to the open source community, but I was still skeptical about its practicality because there would be much more necessary software modules such as recording the TS (transport stream) and gathering EPG (electronic program guide) than just ARIB25 decryption. In the summer of 2010, my continuous net-surfing finally rewarded me to find an excellent blog site that thoroughly explains how to build a digital TV recording system in a plain language with practical examples. This definitely ignited me! Though the site provides only the information about how to get the necessary software modules that a tech-savvy person living in Japan will find useful for building his own system, but it never offers a turn-key solution or a one-command-for-all package, it was indeed a great help for me! I posted a few additional questions to the site, and the site owner was kind enough to provide all the necessary information. So, I decided to start a new project.

Proceed to Digital System Hardware.

Analog System Software Architecture

Here's a link to the downloadable tarball. Please note that there are two [Mm]akefiles, "Makefile" and "makefile" with different cases, in the  same directory level. Operating systems with a case-insensitive file system such as Windows may get confused.

Here are some details about the software architecture.

The software roughly consists of the following components.

• Program recording scheduler with GUI
• Program recording executive
• Recorded program transfer over the Internet
• Watching recorded programs on a couch

All GUIs are implemented as simple web pages. My HTML skills have stayed in the pre-CSS/Flash era forever, and HTML 3.0 level coding is the best I can do!

The majority of the recording scheduler consists of PHP scripts except a few written in Perl. The Perl wrapper program tv.cgi works as a "filtering proxy" for displaying a commercial TV program site on the right pane of the GUI. tv.cgi prepends every <A HREF="..."> tag found in the displaying TV guide content with its own URL. This makes clicking the link be redirected to tv.cgi itself. tv.cgi reads from the passed URL and checks its MIME type. If the MIME type is iEPG, it spawns a PHP script for scheduling; for all other MIME types, the contents are passed through with the aforementioned URL modification. This is how my system can use arbitrary commercial TV guide sites for getting iEPG. If a link is buried in JavaScript, it's very difficult to correctly parse and modify its URL, but so far tv.cgi appears to be doing its job very well. Below is an example of the modification of a link tag in a content read from a fake URL "http://tv.foo.com/guide/show.php?date=20110105&prog=12345678":

Original:

<href a="iepg.php?date=20110105&prog=12345678">iEPG</a>

Modified:

<href a="http://dvr.mydomain.com/cgi-bin/tv.cgi?u=http:%2f%2ftv.foo.com/guide/iepg.php%3fdate=20110105%26prog=12345678">iEPG</a>

A scheduled program's information (iEPG) is stored in a simple database where each record (scheduled TV program) is saved in its own file. The scheduler PHP script spawns a process of a Perl script start-recording, the recording executive, per each scheduled program. start-recording declares its ownership of the scheduled program and sleeps until the program's start time found in the passed iEPG. The Hauppauge video device (/dev/video[01]) is run by the ivtv driver that outputs an MPEG2 stream. start-recording simply spawns a cat process to copy the stream from the device to a disk file. At the end of the program time, start-recording simply kills the cat process to terminate recording, then spawns ffmpeg to transcode the MPEG2 stream file to an AVI file with bit-rate reduction (8Mbps -> 2Mbps) for faster transfer and for compatibility with the viewing equipment (Buffalo Network Media Player). Actually, start-recording could use ffmpeg instead of cat to directly transcode the Hauppauge's output stream, but my system's CPU is unfortunately not powerful enough to transconde the input stream in real time so that the spooling step is necessary. More recent and powerful CPUs are supposed to be able to handle direct transcoding.

A cron'ed daemon process periodically looks for an orphaned scheduled program recording to cover up accidental death of its owner start-recording process. This is a protection against a system reboot and other unpredictable incidents. The daemon process is also responsible for projecting regularly (daily, weekly) scheduled programs to individual ones with validation against their real iEPG, deleting aged recordings to reserve enough free disk space for new recordings, and other clean-up jobs.

A newly recorded program's AVI file is marked "ready." The local home server in California periodically polls a URL on the recording server looking for a new "ready" program. The home server then downloads the ready AVI file and its iEPG file using rsync over ssh. rsync is very useful as it can resume a half-way download interrupted by any reason and as it can transfer files of any size, whereas Perl and PHP on a 32-bit system cannot handle files larger than 2GB. Since the number-crunching ffmpeg transcoding and rsync file transfer are very resource-hogging jobs, they employ a one-at-a-time-and-shortest-first policy to make as many recorded programs viewable as quickly as possible. A very recent change in my code has made it possible to run as many ffmpeg processes as the number of CPU cores (including hyper-threading), but my existing system has only a single-threading CPU...

A downloaded AVI file is played back by wizd on a Buffalo network media player. A set of PHP scripts on the local home server manages recorded TV program files (delete, archive, etc).

That's it!

Back to the latest updates.

Analog System Specification

Here's the specification of the system.

General

Design goals	Watch TV programs broadcast in Japan on a large screen TV in family room in the U.S. Remotely and securely control everything over the Internet Use of and connectivity with open source resources.
Installation	December 2005
Availability	99.9% (actual, excluding scheduled down-times)
Base recording server installed in downtown Tokyo
CPU	AMD Sempron(tm) Processor 3000+ (1.8GHz)
Memory	512MB (64MB used for VGA) DDR3200
Disks	100GB SATA/150 (system software and recorded TV programs) 750GB SATA/150 (local and remote backups)
TV capture cards	Hauppauge PVR350 with hardware MPEG2 encoding (2 ea.)
UPS	A dumb UPS (no communication with server) prevents the server from power-cycling at a short period of power outage
Hardware cost	~ $600US
OS	Linux 2.6 (Fedora Core 4)
Control and processing software	Various open source tools including ffmpeg Home-brew programs written in PHP, Perl, and Bash
Recording
Scheduling	iEPG with point-and-click on a commercial TV guide site using URL-wrapping technology
Regular programs	Day of week, Monday-Friday
TV stations	VHF and UHF analog (1-12ch, 46ch)
Simultaneous recording	Up to 2 programs with automatic arbitration of TV capture cards
Primary recording	MPEG2 720x480i 29.97fps NTSC 8Mb/s (3.9GB/h with audio)
Transcoding	MPEG4 640x480i 29.97fps NTSC 2Mb/s (966MB/h with audio)
Audio	Monaural, 128kbps after transcoding
Transcoding performance	About 220% of program time (software transcoding with ffmpeg)
Simultaneous transcoding	A significantly shorter program can cut in with a higher priority (the number of simultaneous transcoding processes cannot exceed the number of software-visible CPU cores plus one)
Recorded program transfer
File transfer	Rsync over SSH (possible to resume interrupted transfers, no file size limit, one program at a time, shortest program first)
Transfer speed	Up to 800KB/s (in average, a 1-hour program transfers in 25min., limited by local cable modem downstream speed)
Internet security	Packet filtering, SSL, server access deny/allow list
Play back
TV	Sony KDF-50WE655 50" LCD RPTV
Network Media Player	Buffalo "Link Theater" PC-P3LAN2/DVD
Resolution	640x480i on 720p screen
NMP server (hardware)	Intel Core i3 3GHz, 4GB memory, 3TB disk (RAID5), Linux 2.6 (Fedora Core 13)
NMP server (software)	Wizd (modified)
Program menu	Date, Time, Program title, Program subtitle, Station, File size
File management	Delete, Archive (web based)
Playback quality	No worse than good VHS

Proceed to Software Architecture.

Analog System Screen Shots

Here are some screen shots of the web interfaces of the recording server and the home central server.

Recording server portal The web interface is compatible with all known commercial TV guide sites. There used to be a few more sites available, namely Yahoo! and Nikkan-Sports, but they have terminated TV guide services. Sadly, goo, my most favorite site for its simple and easy views, has stopped providing iEPG in its site, so it's no longer usable.
Busy recording server
Now Rec'ing Recording in progress Waiting Waiting to be transcoded Transcoding Transcoding in progress Ready Ready for downloading Xferring Downloading in progress
Scheduling a new recording and canceling a recording
The left pane is an interface to the recording server's file-based database (described later). The right pane shows a commercial TV guide site or the scheduled recording chart. Clicking on an "iEPG" icon will spawn a simple dialog over the clock/calendar pane for confirmation of a recording (above left). Canceling an already scheduled recording is to click a link on the schedule chart (above right) or a link in the schedule table (below left).
Scheduled (but not yet started) programs and regular programs
Scheduled regular (daily, weekly) programs are periodically scanned and projected as individual recordings. Each scheduled recording is periodically checked against its latest iEPG for updating the program title, program subtitle, casts, start time, and end time.
Recorded (and transferred) programs on Wizd NMP server
A Buffalo PC-P3LAN2/DVD network media player plays back recorded TV programs stored in a Linux-based home server. Program selection is point-and-click using an IR remote control. A modified version of the Wizd media server program hides cryptic internal file names and displays program information retrieved from iEPG meta-data instead.
Playing back a recorded program on a 50" LCD RPTV
TV programs recorded at a 480i resolution are played back in 720p mode. The 720p resolution was chosen to display as much program information as possible with reasonable legibility on a large TV screen (see above). Quality is close to that of a good VHS VCR, with no jitter noise at all! A simple survey showed that each week the server records and transfers an average of 45 TV programs totaling over 22 hours (approximately 23GB).
Recorded file management
A web interface running on the local home server manages recorded programs to (delete or archive).

Proceed to Specification.