For TV Tech Leaders, Total Cost Of Ownership Rises To Top Concern

Virtualization, cloud and AI were all hot topics among broadcast technologists gathered at TVNewsCheck’s Local TV Strategies conference at NAB Show New York last week. But by far, the most important theme was “TCO,” or Total Cost of Ownership, a financial estimate that takes into account not only the cost of acquiring a new piece of technology but also the expenses of operating, maintaining and eventually disposing of it at the end of its useful life.

TCO has become an essential metric as cash-strapped station groups carefully weigh technology investments against a declining market for traditional broadcast advertising. And it came up repeatedly during the panel discussion “Technology and the TV Station Group’s Future,” moderated by this reporter, where broadcasters and vendors discussed how stations can use new technology to remain competitive in an increasingly multiplatform world.

Fox’s Cost Focus

Like many groups, Fox Television Stations has shrunk the technology footprint at its stations and improved redundancy by virtualizing many functions as software applications running on common off-the-shelf hardware (COTS).

“That includes the ability to recover quickly, the ability to patch our systems quickly, the ability to spin up new technologies more quickly,” said Paul Capizzi, Fox Corp.CIO and SVP of enterprise technology operations,

The next step is moving key broadcast workflows into the public cloud, something the Fox network has already done with its new operations center in Tempe, Ariz. But with a “heavy focus” on cost, the station group is keeping most things on-prem for now.

“I bring it back to the cost, because I think the costs are always the most challenging,” Capizzi said. “For years, we thought about moving things to the cloud quickly, but then the more we dug into it we realized, ‘OK, I think on-prem right now is more stable. Let’s shrink our footprint. Let’s build a foundation that’s on-prem that we could recover, and leverage the cloud for the resiliency, the backup, the recovery, in the event that we needed to.’”

Fox didn’t suffer any major issues from the major outage that Amazon Web Services suffered on Oct. 20. Neither did Sinclair, which has moved many key workflows into the AWS cloud and is currently relying on the cloud for playout at more than 100 of its 185 stations using Amagi software.

“We had no significant problems,” said Sinclair SVP-CTO Mike Kralec.

That point was reiterated by Amagi EVP of Global Sales Dan Marshall, who cited Sinclair’s use of multiple AWS zones for redundancy.

“You can build infrastructure in the cloud to stay up and running, even during these outages, like we’ve seen through our customers,” Marshall said.

But the AWS outage caused Capizzi enough concern that he would prefer to take a “multi-cloud” approach if the Fox stations were to expand their use of the public cloud, “so we’re not so dependent on one technology or one platform.”

Refining On-Prem Workflows At Nexstar

The possibility of a major outage is one reason Nexstar has decided to keep its key workflows on-prem, said Nexstar EVP, Operations Blake Russell. But a bigger one is that the last time Nexstar ran the numbers, it estimated that moving to cloud-based master control would cost three times as much as an on-prem solution. Russell acknowledged the math might be different for other groups.

“Everybody is approaching things in a little bit of a different way,” Russell said. “We think of ourselves as technologists, but at the beginning and the end of the day, we’re businesspeople, and we’ve got to respect that balance sheet. I happen to represent a company that is really not interested in raising operating expenses.”

However, Nexstar was very interested in streamlining its operations, particularly as it faced the challenge of integrating different technology stacks it inherited with the acquisitions of Media General and Tribune. At one point around 2020 it had 12 master control hubs and a lot of “homegrown” technology from Tribune that it wanted to replace.

“We decided that we were going to come down to five ‘super hubs’ across the United States,” Russell said. “And so, we built these super hubs out, and we sort of pride ourselves in having what we believe is the lowest operator-to-insertion channel [ratio] in the industry at this point in time. And we continue to put our own safeguards in place for controlling content and being able to back up different places along the way. If we have one particular ingest operation go down, we can immediately switch over to another one.”

Cloud Flexibility For Sinclair

For Sinclair, migrating its content ingest, prep, distribution and playout workflows to the cloud over the past several years wasn’t simply about moving off on-prem hardware to cloud instances, Kralec said. Instead, it was “about having opportunity and optionality in our technical ecosystem” to support innovation across Sinclair’s business.

As part of the move to cloud playout, Sinclair has tweaked its own hubbing model, moving from location-based hubs that ran master control operations for multiple stations across a certain geographical area to “media operations centers” with operators focused on running cloud-based workflows for one particular network affiliation across multiple stations.

If the same level of flexibility could be afforded today by hybrid or on-prem architectures, Kralec said, then Sinclair would have gone that route instead.

“It’s not about necessarily technology, as much as it is what that technology unlocks in terms of opportunity,” Kralec said.

For example, since launching in the cloud Sinclair has been through four different processor generations inside of AWS, each one more efficient than the last. And it was able to use the data and analytics it gets from its cloud tools, along with AWS Step Functions (an orchestration tool) and Elemental media services, to completely overhaul its transcoding workflow in three weeks.

“This is about the rate of change inside of all of our organizations, and how quickly we need to adapt as technologists to what the business is asking for while maintaining the high-quality broadcast that we expect,” Kralec said. “It’s really about being able to leverage all our technology. And like I said, it is not a specific technology. It’s not necessarily cloud. It’s about architectures that don’t tie into a particular operating model for a long time.

“And getting back to the financial piece of this, what it unlocks, all those opportunities, means that we now unlock looking at technology through a financial prioritization lens,” he continued. “And that means taking the value and the risk into every technology decision and prioritizing what we want to do.”

TCO, POCs For Hearst

Hearst Television’s technology infrastructure is still mostly on-prem, said Stefan Hadl, SVP, broadcast engineering and technology, Hearst Television. But the group has already virtualized a lot of its workflows and uses the cloud for some tasks. Like the Fox stations, Hearst is now evaluating whether it makes sense to move some of its virtualized workflows into the cloud.

“TCO is going to drive whether we make that move,” Hadl said. “Reliability and quality is job one. As a local broadcaster, for Hearst Television, our commitment is to our communities and making sure that we’re there when they need us.”

Hearst is on its third generation of virtualization software and has been testing potential replacements. The group also has aging HD-SDI infrastructure at some stations that is due for replacement soon, and so it has conducted POCs (proofs-of-concept) of 2110 routing technology to learn about its feasibility.

“For us, in the POCs that we’ve done with 2110 at two of our locations, we’re seeing the benefit of infrastructure,” Hadl said. “The infrastructure that it takes to wire up and cable and integrate systems in 2110, although it’s a heavy learning curve for your support team, the infrastructure really shrinks. That’s the biggest takeaway with the POC that we did, is that wait a minute, all these sources now are in a 2110 environment that can go anywhere, anywhere, anytime. I don’t have a point-to-point coax cable that I need to get the signal around my facility.

“That’s where we’re seeing the leverage and the savings where you offset the cost of a red/blue [IP switching] network to make sure you have the resiliency and the redundancy that you need to make sure you’re up 24/7,” he continued. “You’re offsetting the cost of the integration to pay for that, as opposed to systems in SDI today.”

On the other hand, Hearst could buy new HD-SDI routers from a vendor like Evertz, and they would probably run for the next 15-18 years, Hadl said, so in some places taking a plant 2110 will make financial sense, while in others it simply won’t.

Hybrid Options From Evertz

“Hybrid” is a broadcast buzzword that generally means a plant with a mix of traditional HD-SDI and 2110 gear, or a combination of on-prem and cloud infrastructure. As a vendor that offers products in each category, Evertz finds that “hybrid has many different contexts,” said Evertz VP of Sales Orest Holyk.

“In the Evertz world, if you’re IP, are you COTS or are you Evertz IP-type switches?” Holyk asked. “Hybrid could be on-prem or off-prem, and there are additional cloud options in there. So that word hybrid, and how it is defined and how it is used by each individual station or station or network or whatever entity it is, is very dependent as to what Evertz is doing and what Evertz’s approach is to providing a solution.”

Holyk said on one level, it’s great that there is still a lot of legacy SDI gear running after 15 to 20 years. But the problem is that support for many of those products is going away. So Evertz is working with customers to help them pick a path forward from that aging gear which still fits their budget. Familiarity with the replacement technology, and the ability to maintain and support it, is a key consideration.

“Because the SI [systems integrator] will get in there, they’ll do the work that they need to do,” Holyk said. “They’ll finish their job, and then they’ll take a step back. And then the station groups are left holding the job, holding that responsibility of maintenance, and making sure that it continues to operate 24/7.”

He said the Fox stations were a good example of a hybrid architecture that Evertz is supporting. Fox is using Evertz Streampro as an on-prem, automated playout solution, while employing the Evertz I/O cloud product as a disaster recovery system.

“They’ve got a hybrid approach that covers all the bases that they need,” Holyk said. “And then the Evertz I/O also allows them to spin up a FAST channel if they need to do that.”

Amagi Pitches Cost Reduction Through Cloud

Amagi’s products are 100% cloud-based, so it isn’t selling hybrid hardware and software solutions like Evertz. Instead, it is pitching the cloud as the best way for stations to reduce costs as they cope with declining ad revenues by getting rid of “technical debt” and eventually trimming headcount. Marshall said that 80% of the RFPs Amagi has seen in the last nine months are for such “broadcast modernizations.”

But the RFPs aren’t just aimed at replacing legacy HD-SDI gear or offloading aging data centers. About 90% of the RFPs also include a managed services component, which has led Amagi to create a new operations center in New Jersey to handle master control and monitoring functions.

What Amagi originally thought would be a two-step process to reducing total cost of ownership is now being contemplated by many broadcasters as one big move, Marshall said.

“We thought that there’d be a two-putt motion,” he said. “Get out of the technical debt first, find out if it makes sense to go to cloud. And then the second component is, does it make sense for us either to manage in-house or push over to managed services someplace else? And we’re seeing that that those are being compressed. RFPs are coming out now and they’re looking at both.

“This is not speculation,” he continued. “This is real broadcasters globally that are going through this, looking at broadcast modernization, driving down total cost of ownership from a capex perspective and from an opex perspective, and using cloud to do that.”

AI: Moving Fast, With Guardrails

All of the broadcasters on the panel said that they were experimenting with AI as a way to streamline operations, in many cases creating their own AI chatbots to help with mundane daily tasks. But they also said that maintaining human supervision of AI workflows was essential, particularly in areas like news production.

Fox is using generative AI across different parts of the company, Capizzi said, with finance using it to build dashboards, HR employing it for job descriptions and software developers using it to help write code. He is looking to expand its use while keeping a close eye on costs.

Kralec and Hadl both said AI can be a very useful tool for managing content, particularly for searching through archives to be able to repurpose footage, something both groups are exploring.

“AI is going to be a huge enabler if it’s used the right way, and we put the right guardrails around it,” Hadl said.

Nexstar has used AI successfully on the creative services side as a way to create quick mock commercials to pitch to prospective advertisers. In some cases, those AI-generated commercials impressed clients so much that they took them straight to air.

But Nexstar has been slower to implement the technology on the news side, said Russell, who sits on an internal committee that vets that application of AI across the company.

“Right now, AI is really sort of the Wild Wild West, and there’s a lot of stuff out there,” Russell said. “From a digital component, we’re trying to figure out how we can take those traditional broadcast stories and turn those into digital verticals at this point within AI. But you can’t just say that’s one-and-done. You’ve still got to have somebody to be able to read through that and make sure that you’ve still got that preservation of the continuity of the story itself. And that is what’s very, very important.”

The post For TV Tech Leaders, Total Cost Of Ownership Rises To Top Concern appeared first on TV News Check.