The video compression landscape will become more crowded than ever in 2020, with the release of three codecs from The Motion Picture Experts Group (MPEG). Versatile Video Coding (VVC), MPEG-5 EVC and LCEVC will enter a market already served by MPEG-2, MPEG-4 AVC (H.264), HEVC, AV1 and a smattering of other niche production and contribution codecs. Deployment decisions are further complicated by the emergence of alternative bitrate saving technologies such as context aware encoding (CAE) and precoding. This may leave even less room for new codecs to emerge and further entrench AVC as the incumbent standard for many years to come.
This current state-of-affairs is not surprising given the complexity of modern media. The sheer number of codecs and compression options reflects the myriad channels and formats through which we consume video. A single asset may require 9 different bitrate versions in its encoding ladder to service the current device ecosystem. The co-existence of legacy, contemporary and emerging codecs could be the new normal, because each codec operates within an optimal quality, resolution and bitrate window. We may never see the homogeneity of MPEG-2 or AVC again because the content and distribution ecosystem has become highly inhomogeneous.
A certain level of homogeneity is in the interests of the industry. It is more efficient in terms of storage and traffic costs and, most importantly, it maintains a consistent and predictable addressable market of consumer devices. What has led to a fragmented compression situation seemingly in contrast with industry interests?
Firstly, the adoption of a codec is a complicated affair. Content creators, operators, device manufacturers, chipset vendors, Video Processing service providers and patent holders must all be on the same page. Any misalignment between these overlapping interests can mire a technology in a web of discord unrelated to its technical merit. The HEVC codec provides an interesting example of how fraught codec development, licensing, and adoption can be. It was introduced in 2013 to usher in the age of UHD content and become the de-facto successor to AVC, but complex and expensive licensing has overshadowed its achievements.
HEVC has found itself on the front line of a power struggle for control of UHD compression. AV1 was developed as a rival codec in 2018; royalty free and with equal technical merit, it promised to solve HEVC’s problems for free. AV1 was created and championed by the Alliance for Open Media, a group founded by Amazon, Apple, Google and Microsoft among others. The broadcast industry, represented by MPEG, interpreted AV1 as a power grab and were loath to relinquish anymore influence in the video space to the tech giants.
MPEG responded head-on by developing MPEG-5 EVC; a codec that employs a freemium model. It has a free profile named Baseline aimed at neutralizing AV1’s value proposition and a premium profile named Main is designed for maximum performance. MPEG-5 is also a tacit acknowledgement that the business model that governed HEVC is broken. MPEG-5 will challenge AV1, but risks undermining HEVC as they offer similar efficiency gains. Samsung, Huawei and Qualcomm have already publicly backed MPEG-5, the tech giants remain firm on AV1, while HEVC has an incumbent’s advantage with a seven-year head start. The future of UHD compression is more uncertain than ever.
Uncertainty about the present has not stopped MPEG from considering the future. The second of the three codecs announced is Versatile Video Coding (VVC). A technology that stakes a claim to the 8K, 360 video, AR and VR genres of the future. It promises impressive technical gains over current codecs, but it’s unclear whether it solves the royalty issues plaguing current technologies.
As the codec wars play out before us, a crop of new technologies to augment current codec performance has quietly emerged which may prove as effective in saving bandwidth as next-gen codecs themselves: namely CAE and precoding. The first is ‘precoding’. A technique that enhances details the viewer is most likely to perceive while simultaneously diminishing details the viewer is likely to ignore. Importantly this technology sits on top of the current codec ecosystem. iSize is a ‘precode’ start-up whose perceptual quality algorithm promises to save 30-50% in bitrates by optimizing frames for maximum efficiency. Second is CAE; a technology that uses analytics and subjective quality metrics to vary bitrates to match quality goals, instead of the traditional method of fixing quality to match bitrate goals. This technology sits on top of current infrastructure and importantly matches the 30-50% gains in efficiency that new codecs, requiring their own ecosystems to function, are promising. Another important facet of these solutions is the application of AI techniques. The subjective quality metrics that underpin these technologies are continuously optimized through machine learning; to the point where optimal encoding parameters can include metrics such as the consumer’s device type or viewing habits. Synamedia, Harmonic, Brightcove, MediaKind, Netflix and Haivision all employ proprietary versions of CAE that use their own subjective quality metrics to save bandwidth.
What does this all mean for the video compression market? There are four key points to consider and they all suggest that AVC’s dominance will continue for a long time.
Firstly, the compression landscape for contribution and distribution remains mostly homogenous. In 2019 71% of compression-related video processing spend was on AVC technologies, 13.1% to MPEG-2 and 10.4% to HEVC with the remainder made up by other codecs. HEVC barely scraping into double digits may seem like a lucklustre share of the market for a codec meant to inherit the mantle of AVC, but it is understandable when considered against its intended purpose to serve UHD content. The real inhibitor to HEVC adoption has been the slower than expected demand for 4K content. This same dynamic will affect AV1 and MPEG-5 regardless of their technical specifications and licensing schema.
Secondly, incumbent compression technologies such as AVC tend to be very sticky due to the installed base of consumer devices that depend on dedicated hardware encoding/decoding. More flexible software-based encoding is becoming increasingly ubiquitous, but the role of dedicated end-device hardware decoding has an outsized influence on a codec’s addressable market.
Thirdly, AVC’s longevity is likely to be compounded by the fact that it was the last codec to be adopted in the good faith that IP licensing would be clarified in a satisfactory manner. While that assumption worked for AVC, it was undermined by HEVC; the fallout from which the industry is unlikely to ever recover from. This makes it highly unlikely that any future codec will be adopted en-masse without iron clad guarantees on cost and licensing. This guarantee is absent in all new codecs except MPEG-5 baseline. Another factor extending longevity is AVC’s adoption coinciding with the industry’s transition to HD and IP. This means the majority of archived content is AVC and archives remain relevant for at least the lifetimes of the generation that created them.
The final point is the role of precoding and content aware technologies. They are in a prime position to provide the bandwidth savings that new codecs are expected to offer without the licensing issues. Where the AVC and HEVC use case overlapped, for 1080p and 2k content for example, the emergence of CAE and precoding has made the choice to stick with AVC a compelling one. The application of HEVC for HD content remains a niche use case. MPEG themselves seem to have recognized this scenario by introducing LCEVC, the third and final of the new codecs. It has more in common with the precoding and CAE technologies discussed above than other codecs. It works by complementing current compression workflows through correcting artifacts and ultimately allows quality goals to be achieved with fewer processing and memory resources. Important to note that it works with any codec.
The choice of underlying codec in a video workflow will factor in cost transparency and addressable market more heavily than in the past. Performance and efficiency will remain key considerations but can increasingly be offloaded to ecosystem agnostic technologies such precoding, CAE and LCEVC. The short to mid-term effect will be to contribute to the longevity of AVC, which is likely to prove a frustratingly stubborn incumbent for new codecs to shift. The long term will undoubtedly produce a true successor to AVC, but only when UHD demand grows and if attempts at simplifying licensing are successful.