On March 19, a violation of computer security procedures at public TV station KLCS in Los Angeles accidentally introduced a self-replicating virus into the station’s editing systems.
Just over six months ago, three workers on an Iowa Public Television tower failed to heed tower safety practices and plunged 1,200 feet to their deaths.
One weekend last summer, an intern at NPR Labs discovered he could easily hack into a software program commonly used in HD radio production.
In October 2002, the chief engineer at KRVS in Lafayette, La., realized his station had no disaster recovery plan after Hurricane Lili tore off the roof and completely soaked all of the equipment.
How could public broadcasters have avoided these security and safety breaches, or at least reduced their consequences? That’s the meat on the menu at the first joint session of public TV’s and public radio’s annual engineering conferences, April 13 [2007] in Las Vegas.
Tackling the virus at KLCS was the assignment of Alan Popkin, director of engineering. Even though his staff was prepared to fight, recovering took four days of work.
“Because we built redundancy and resilience into our systems, we isolated the infection pretty quickly,” says Popkin, who sounded worn but chipper after four days of intensive virus-busting. “Since we locked the virus out of our on-air equipment, viewers weren’t affected.”
“Still, we had to clean every single machine in the automation network to ensure we’d eradicated the threat,” Popkin says. “It took us 8½ hours to clean the editing system alone.”
Matt Burrough, who will kick off the security roundtable at the Las Vegas conference, also put in some long hours on a different security issue. Fortunately, he was working to prevent an incursion rather than recover from one.
Burrough, a Rochester Institute of Technology student who interned at NPR Labs last summer, realized he wasn’t seeing enough security to rebuff threats the network might face.
The next day, Burrough conducted a preliminary investigation. “I ended up with nine pages of concerns,” he says. “I passed them around the office and, eventually, addressing them also became my senior thesis for RIT.”
Among other things, Burrough discovered significant weaknesses in the program associated data (PAD) software developed by iBiquity Digital Corp. for HD Radio broadcasting. “The PAD software allows for displaying an audio feed’s creative information on HD radios,” explains Burrough. “This includes artist, title, recording label, etc.”
Burrough, who will join NPR Labs full-time upon graduation, is a self-avowed “non-hacker.” He uncovered PAD’s flaws by studying vulnerability lists on the Web. Then, he used a separate PC to test the security of the PAD computer. “I interrupted the correct information and substituted my own,” he says. “So, potentially, a hacker half way around the world could put any type of text into an improperly protected PAD stream.”
Not surprisingly, there’s more. “It’s appears theoretically possible to hack into multicast feeds,” he warns. “I just haven’t tested my theory yet.”
Meanwhile, NPR Labs is mitigating security shortcomings by physically separating production systems from desktop computers that use office software or interact with the Internet.
In other words, securing computer systems goes beyond addressing specific software vulnerabilities, such as those in iBiquity’s PAD. “With computer security you can never say, ‘OK, this is it—if we fix this problem, we’re done,’” Burrough stresses. “You must address security systemwide and develop appropriate intervention measures.”
Co-panelist Ken Walters, senior director of enterprise platforms at PBS, agrees. “IT technology-based broadcasting brings a whole new set of challenges.”
In particular, security and security practices will always be evolving, says Walters. What’s more, there’s no one-size-fits-all solution. “Basic standards and security practices do exist,” Walters says. “But, customers and vendors need to seek out and apply them. Also, each organization must decide what layers of security work best for their situation.”
Of particular concern for pubcasting is its reliance on technology from many small, boutique vendors. Throughout computing history, such firms haven’t always incorporated the latest security measures in their products. Typically, such products haven’t had robust security until users insist on it.
“Public broadcasters must start ensuring that vendors have made secure products before they complete purchases,” stresses Walters. “This includes, but isn’t limited to, compatibility with current anti-virus solutions, the application of critical operating system patches and generally more robust systems designed for complex computing environments.
Karl Fontenot, chief engineer at KRVS in Lafayette, La., speaking in Las Vegas, will cover what he learned from hurricanes Lili, Katrina and Rita, plus lessons drawn from the Katrina experiences of his counterpart at WWNO in New Orleans.
Fontenot can’t recall hurricanes ever forcing dwellers of the inland city of Lafayette to evacuate—until Lili.
After the storm passed, Fontenot discovered the roof was gone from the station’s sheet-metal building. Fontenot and his g.m. covered the structure temporarily. “Then I dried each piece of equipment, individually with a hair dryer, using our 5 kilowatt generator,” Fontenot recalls.
Although restoration tasks were tedious, Fontenot says the greatest stress was caused by not knowing who was available to help and who needed help. KRVS had no basic plan for communicating during and after a major emergency. The situation was complicated by lack of power and telephones for four days. Plus, the powerful storm had ripped massive trees out of the ground, making travel tenuous at best. “Some of our key employees didn’t have cell phones at that time,” explains Fontenot. “And, they weren’t where we expected, because they’d re-evacuated as the storm intensified.”
“Not knowing where our people were, or whether they were safe, created unnecessary worry and stress,” continues Fontenot. “Although everyone was eventually found, Lili caused us to re-evaluate and begin establishing better, more formalized, plans. You need to designate who will be in charge of which tasks, how people will communicate or where rendezvous locations will be depending on the final direction of the storm.”
A significant capital item in Fontenot’s new plan is a 175-kilowatt generator and related infrastructure. “Prior to Katrina and Rita, I was budgeting $63,000,” Fontenot explains. “Now, the same system is almost $225,000 because disaster preparedness infrastructure costs, including installation, have skyrocketed all along the Gulf Coast. I’m still figuring out whether splitting the project into pieces will make it affordable by spreading the cost over several years.”
Downstate in New Orleans, many of WWNO’s dramatic Katrina tales reinforce the value of simple preparedness measures. If the station had taken one particular precaution, for instance, it could have returned to the air in hours. Instead, it took 21 days.
“Before Katrina, our transmitter cable was strapped to the outside of our tower, which is quite common,” says Robert Carroll, WWNO’s chief engineer. “Since our studio equipment was on the fourth floor, it stayed dry, so we could have been operational immediately after the storm. But when I looked up at our tower, I saw 150 feet of transmission line had broken off and was wrapped tightly around a guy-wire, so I knew that was impossible.”
As WWNO continues fortifying its University of New Orleans site so that it could remain on the air through hurricanes as severe as Category 3, the scope of Katrina’s devastation leaves many questions.
“We’ve been considering full failover locations for our computers,” says Carroll. “With Katrina taking out 120 miles of coastline and a similar swath all the way north to Jackson, a full-time failover site that’s far enough away would be prohibitively expensive.” He’s still searching for a solution.
If there’s good news, Fontenot and Carroll say, it’s that hurricanes have raised awareness of their stations’ public service value. “Previously, we were considered ‘just university stations,’” Fontenot comments. “Now, we’re viewed as a vital point of contact, particularly by public health and safety officials.”
Carroll notes that the FCC has already started preliminary discussions about requiring stations across the country to have full disaster plans on file.
Although pubcasters outside storm and earthquake zones may think they can skip comprehensive disaster planning, Chief Engineer Bob Wyatt of KSPS in Spokane, Wash., reminds everyone that a simple cold snap may cost his station up to $1.3 million. As Current reported last December, about a third of the station’s tower snapped off during Thanksgiving weekend.
“There wasn’t any wind or ice buildup,” says Wyatt. “Since temperatures were in the single digits, the theory is the 600-foot guy wires contracted significantly, putting stress on the metal anchors, one of which gave way.” After the accident, initial tests showed the failed anchor had a previously undetectable crack. Further tests revealed three more of the 36 anchors were also cracked.
Eventually, the station mounted an antenna on the 400-foot remainder of the tower, Wyatt says. “But, the tower was built in 1967 and we’re still waiting to see if the remaining section is sound.”
Of course, it’s not just age or weather decay that will cause towers to demand attention. According to Wyatt, some studies suggest as many as 600 towers, coast to coast, will require servicing as stations add digital transmitters. ”And, there’s only seven certified tower-climbing firms in the nation,” Wyatt says.
Tragically, the tower conversion crunch isn’t what prompts the presentation on tower safety at the TV/radio session in Las Vegas. Instead, it was a fatal human error by one of the most safety-conscious climbers in the business, Leo Deters.
Last September, Deters and two employees plunged 1,200 feet to their deaths from an Iowa PTV tower near Des Moines. In violation of state safety codes, the team was riding up the tower on a winch instead of a proper platform or elevator, when a cable broke. The family-run Deters Tower Service was fined for safety violations and put up for sale.
“We’ve all made mistakes we’ve gotten away with, but Leo didn’t,” says his long-time friend, electrical engineer Tom Silliman, with trademark directness. “Frankly, you’re as dead if you fall 100 feet as if you fall 1,000—you just have more time to think about it.”
Silliman, president of tower engineering firm Electronics Research Inc., of Chandler, Ind., and a board member of WNIN-FM/TV in Evansville, is revisiting his tower safety presentation originally given at last October’s Iowa DTV Symposium. He realized last year there were no materials to provide stations with guidance for working with tower crews.
“In fact, many climbing and construction standards themselves are less than five years old, yet much of the tower building and servicing equipment is decades old and homemade,” says Silliman, whose company designs and manufactures towers, antennas and related equipment. “I agreed to write the presentation so broadcasters would know what specifications and procedures they should insist on before anyone sets foot on a tower.”
Silliman’s presentation, which he dedicated to Deters, received top honors at the DTV Symposium and remains the only resource of its kind. If there’s one chief engineer likely to heed Silliman’s advice its IPTV’s Bill Hayes.
“I wasn’t at the tower that day and I still wake up nights thinking, ‘What if there had been an elevator?’” says Hayes. “Since the accident, we’ve abandoned plans to lower operating expenses by removing elevators from two other towers. It just isn’t worth it.”
Ultimately, broadcasters can’t prevent freak accidents, Hayes says. “But, if you know the requirements from [Silliman’s] presentation, you can at least ask a tower crew to show you their certifications and their climbing plan,” he says. “If they can’t, maybe you don’t want them on your tower.”
“Remember, if you’re not digital on Feb. 18, 2009, the FCC isn’t going to say, ‘Send a bunch of cowboys up your tower or we’ll shut you down,’” Hayes asserts. “Sure, you could get a lecture and a fine. But, nobody will be required to send unqualified workers up your tower to complete transition tasks because nobody should have to give their life for the sake of our business.”
Anne Rawland Gabriel, technology editor of Current, covers the PBS and NPR technology conferences and the National Association of Broadcasters Convention in Las Vegas.
Web page posted April 19, 2007
Copyright 2007 by Current Publishing Committee
Current: the newspaper about public TV and radio
in the United States
Current LLC,
Takoma Park, Md.
301-270-7240
E-mail:
