rtmp2rtc: Support HEVC #4289

duiniuluantanqin · 2025-02-14T06:59:01Z

1. Usage

Launch Chrome with H.265 enabled:

# Windows: 
C:\Program Files\Google\Chrome\Application>"C:\Program Files\Google\Chrome\Application\chrome.exe" --enable-features=WebRtcAllowH265Receive --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

# macOS:
open -a "Google Chrome" --args --enable-features=WebRtcAllowH265Receive --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

Launch SRS with `rtmp2rtc.conf`

./objs/srs -c conf/rtmp2rtc.conf

Push H.265 with RTMP

ffmpeg -stream_loop -1 -re -i input.mp4 -c:v libx265 -preset fast -b:v 2000k -maxrate 2000k -bufsize 4000k -bf 0 -c:a aac -b:a 128k -ar 44100 -ac 2 -f flv rtmp://localhost/live/livestream

Play with WebRTC

http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream

2. Parameter Combinations for SDP

sendrecv offer

--enable-features=WebRtcAllowH265Send,PlatformHEVCEncoderSupport,WebRtcAllowH265Receive --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

sendonly offer

--enable-features=WebRtcAllowH265Send,PlatformHEVCEncoderSupport

recvonly offer

--enable-features=WebRtcAllowH265Receive --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

3. How to test if H.265 is working

Browser Test for supporting H265

https://webrtc.github.io/samples/src/content/peerconnection/change-codecs/

How to test Safari: HEVC: webrtc support hevc on safari. v6.0.34 #3441
Debug in Safari

trunk/src/kernel/srs_kernel_rtc_rtp.hpp

trunk/src/kernel/srs_kernel_rtc_rtp.cpp

suzp1984 · 2025-02-14T15:31:55Z

trunk/src/kernel/srs_kernel_rtc_rtp.hpp

@@ -254,8 +258,10 @@ enum SrsRtspPacketPayloadType
    SrsRtspPacketPayloadTypeRaw,
    SrsRtspPacketPayloadTypeFUA2,
    SrsRtspPacketPayloadTypeFUA,
+    SrsRtspPacketPayloadTypeFUAHevc,


Is the *Rtsp* part of the removed RTSP related code? Is it should be renamed to SrsRtpPacketPayloadType?

maybe it is a typo introduced by @winlinvip

trunk/src/kernel/srs_kernel_rtc_rtp.cpp

trunk/src/app/srs_app_rtc_source.cpp

trunk/src/kernel/srs_kernel_codec.hpp

trunk/src/app/srs_app_rtc_conn.cpp

suzp1984 · 2025-02-14T16:23:57Z

trunk/src/app/srs_app_rtc_conn.cpp

+    return false;
+}
+
+bool srs_sdp_has_h265_profile(const SrsSdp& sdp, const string& profile)


this is a so general method, the profile-id defined in h.265 standard, but just main profile(1), main 10 profile(2), and main still picture profile(3) are commonly used.

why not add more specific wrapper method. e.g. srs_sdp_has_h265_main_profile(const SrsSdp& sdp),
srs_sdp_has_h265_main_10_profile(sdp) and srs_sdp_has_h265_main_still_picture_profile(sdp).

To align with H264, and we can improve it in another PR.

trunk/src/app/srs_app_rtc_source.cpp

trunk/src/kernel/srs_kernel_rtc_rtp.cpp

trunk/src/app/srs_app_rtc_source.cpp

suzp1984

About the SDP parameter and codec name case sensitive issues.

suzp1984 · 2025-02-17T06:48:10Z

trunk/src/app/srs_app_rtc_source.cpp

+        format_specific_param << "level-id=" << h265_param_.level_id;
+    }
+    if (!h265_param_.profile_id.empty()) {
+        format_specific_param << ";profile-id=" << h265_param_.profile_id;


if the level_id is empty, the format_specific_param will output with ;` ahead.
Is this case possible?

Yes, H264 has the same issue. However, calling set_h265_param_desc ensures that level_id is not null.

suzp1984 · 2025-02-17T07:02:54Z

trunk/src/app/srs_app_rtc_source.cpp

+        if (kv.size() != 2) {
+            return srs_error_new(ERROR_RTC_SDP_DECODE, "invalid h265 param=%s", attribute.c_str());
+        }
+        if (kv[0] == "level-id") {


I didn't read the SDP spec, but only asked the AI whether the SDP parameters case sensitive?

It said the SDP parameters are generally case-insensitive. But the best practice is to use consistent casing, so it has some conventions there, e.g.

a=rtpmap:96 H264/90000 a=fmtp:96 profile-level-id=42e01f; packetization-mode=1

H264 and h264 are equivalent. (srs already handle this, I do think so)

profile-level-id and PROFILE-LEVEL-ID are equivalent. (srs not handle this case here at least).

the whitespace between parameters. (e.g. profile-level-id=42e01f;_packetization-mode=1).

It appears the protocol does not explicitly state whether spaces are allowed, but based on experience, they are not used.

suzp1984 · 2025-02-17T07:20:15Z

trunk/src/app/srs_app_rtc_conn.cpp


-            // TODO: FIXME: pick up a profile for HEVC.
-            // @see https://www.rfc-editor.org/rfc/rfc7798#section-7.2.1
+                track_descs = source->get_track_desc("video", "H265");


About this method get_track_desc, I would suggest for refactor later.

there are only 3 kinds of track types audio, video, data, the SRS only has 2 of them, audio/video. A enum is much better here.

Inside get_track_desc, there is a hide precondition of the media_name, the audio codec name is lower casing, while the video codec name is upper case. This method did the force case transform for the stored media, but forget do same transform for the media_name parameter.

That's true. I've also been affected by this when doing case conversions. We can solve this problem with another PR

chundonglinlin · 2025-02-19T13:08:42Z

Currently Safari supports rtc2rtc. We should test the performance in Safari browser. I haven't run rtmp2rtc in Safari yet.

suzp1984 · 2025-02-20T02:24:12Z

Case 1:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc, (I used safari), the SRS will gives answer offer with h265 codec.
But if publish RTC by &codec=h265, the SRS will gives me h264 as offer answer.
Expect: improve WHIP/WHEP api to more flexible. (recognize both hevc, h265, h.265?)

Case 2:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc (safari in my case),
Play the RTC by WHEP: http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream (chrome in my case), H.265 streams play in web browser well.
Then I play rtmp bridged stream: ffplay rmtp://localhost/live/livestream, it plays well too, but with h.264 codec shows in the ffprobe rtmp://localhost/live/livestream.

So I think the rtc to rtmp bridge not consider this case well.

Case 3:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc (safari)
Play RTC by WHEP: http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream&codec=h264 (chrome)
The WHEP api return the answer offer without video media track, and chrome also didn't play the audio also.
Expect: WHEP api should return error code in this case.

chundonglinlin · 2025-02-20T02:31:37Z

Case 1:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc, (I used safari), the SRS will gives answer offer with h265 codec.

But if publish RTC by &codec=h265, the SRS will gives me h264 as offer answer.
Expect: improve WHIP/WHEP api to more flexible. (recognize both hevc, h265, h.265?)

Case 2:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc (safari in my case),

Play the RTC by WHEP: http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream (chrome in my case), H.265 streams play in web browser well.

Then I play rtmp bridged stream: ffplay rmtp://localhost/live/livestream, it plays well too, but with h.264 codec shows in the ffprobe rtmp://localhost/live/livestream.

So I think the rtc to rtmp bridge not consider this case well.

Case 3:

Publish RTC by WHIP: http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc (safari)

Play RTC by WHEP: http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream&codec=h264 (chrome)

The WHEP api return the answer offer without video media track, and chrome also didn't play the audio also.
Expect: WHEP api should return error code in this case.

I think this PR only supports hevc by rtmp2rtc, hevc by rtc2rtc can be fixed in the new PR.

duiniuluantanqin · 2025-02-20T05:41:46Z

@suzp1984 rtc to rtmp is NOT support by now, I will start working on this after the PR is merged.

chundonglinlin · 2025-02-21T08:54:11Z

Currently Safari supports rtc2rtc. We should test the performance in Safari browser. I haven't run rtmp2rtc in Safari yet.

We should test WebRTC 265 on Safari 18.0 and above.
Source: WebKit Features in Safari 18.0 - WebRTC (https://webkit.org/blog/15865/webkit-features-in-safari-18-0/#webrtc)

TRANS_BY_GPT4

chundonglinlin · 2025-02-21T08:56:30Z

trunk/src/app/srs_app_rtc_source.cpp

@@ -1083,6 +1084,10 @@ srs_error_t SrsRtcRtpBuilder::on_video(SrsSharedPtrMessage* msg)
    if (has_idr) {
        SrsUniquePtr<SrsRtpPacket> pkt(new SrsRtpPacket());

+        if ((err = bridge_->update_codec(format->vcodec->id)) != srs_success) {


Each keyframe invokes the source_->get_track_desc method, which could be moved to the initialization section.

TRANS_BY_GPT4

duiniuluantanqin requested review from chundonglinlin, winlinvip and suzp1984 and removed request for chundonglinlin and winlinvip February 14, 2025 06:59

winlinvip added the EnglishNative This issue is conveyed exclusively in English. label Feb 14, 2025

duiniuluantanqin requested review from winlinvip and chundonglinlin February 14, 2025 06:59

duiniuluantanqin changed the title ~~RTC: Support HEVC~~ rtmp2rtc: Support HEVC Feb 14, 2025

suzp1984 reviewed Feb 14, 2025

View reviewed changes

trunk/src/app/srs_app_rtc_conn.cpp Show resolved Hide resolved

suzp1984 reviewed Feb 14, 2025

View reviewed changes

suzp1984 reviewed Feb 15, 2025

View reviewed changes

trunk/src/app/srs_app_rtc_source.cpp Outdated Show resolved Hide resolved

suzp1984 reviewed Feb 15, 2025

View reviewed changes

trunk/src/kernel/srs_kernel_rtc_rtp.cpp Outdated Show resolved Hide resolved

suzp1984 reviewed Feb 15, 2025

View reviewed changes

trunk/src/kernel/srs_kernel_rtc_rtp.cpp Outdated Show resolved Hide resolved

suzp1984 reviewed Feb 15, 2025

View reviewed changes

trunk/src/app/srs_app_rtc_source.cpp Show resolved Hide resolved

rtmp2rtc: support hevc

1a224c7

duiniuluantanqin force-pushed the hevc_over_rtc branch from fc9fa0a to 1a224c7 Compare February 16, 2025 01:19

duiniuluantanqin requested review from xiaozhihong and suzp1984 February 17, 2025 02:23

suzp1984 reviewed Feb 17, 2025

View reviewed changes

duiniuluantanqin added 2 commits February 19, 2025 15:40

remove unused code

b3c5085

define SrsAvcNaluTypeParse

83d577a

Merge branch 'develop' into hevc_over_rtc

c11afbe

suzp1984 approved these changes Feb 20, 2025

View reviewed changes

Merge branch 'develop' into hevc_over_rtc

83e8877

chundonglinlin reviewed Feb 21, 2025

View reviewed changes

Optimize init h265 track and format.

8953dbf

chundonglinlin force-pushed the hevc_over_rtc branch from cd7598a to 8953dbf Compare February 21, 2025 15:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rtmp2rtc: Support HEVC #4289

rtmp2rtc: Support HEVC #4289

duiniuluantanqin commented Feb 14, 2025 •

edited

Loading

suzp1984 Feb 14, 2025

duiniuluantanqin Feb 15, 2025

suzp1984 Feb 14, 2025

duiniuluantanqin Feb 16, 2025

suzp1984 left a comment

suzp1984 Feb 17, 2025

duiniuluantanqin Feb 17, 2025 •

edited

Loading

suzp1984 Feb 17, 2025

duiniuluantanqin Feb 17, 2025

suzp1984 Feb 17, 2025

duiniuluantanqin Feb 17, 2025

chundonglinlin commented Feb 19, 2025

suzp1984 commented Feb 20, 2025

chundonglinlin commented Feb 20, 2025

Case 1:

Case 2:

Case 3:

duiniuluantanqin commented Feb 20, 2025

chundonglinlin commented Feb 21, 2025 •

edited by winlinvip

Loading

chundonglinlin Feb 21, 2025 •

edited by winlinvip

Loading

rtmp2rtc: Support HEVC #4289

Are you sure you want to change the base?

rtmp2rtc: Support HEVC #4289

Conversation

duiniuluantanqin commented Feb 14, 2025 • edited Loading

1. Usage

Launch Chrome with H.265 enabled:

Launch SRS with rtmp2rtc.conf

Push H.265 with RTMP

Play with WebRTC

2. Parameter Combinations for SDP

3. How to test if H.265 is working

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suzp1984 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duiniuluantanqin Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chundonglinlin commented Feb 19, 2025

suzp1984 commented Feb 20, 2025

Case 1:

Case 2:

Case 3:

chundonglinlin commented Feb 20, 2025

Case 1:

Case 2:

Case 3:

duiniuluantanqin commented Feb 20, 2025

chundonglinlin commented Feb 21, 2025 • edited by winlinvip Loading

chundonglinlin Feb 21, 2025 • edited by winlinvip Loading

Choose a reason for hiding this comment

duiniuluantanqin commented Feb 14, 2025 •

edited

Loading

Launch SRS with `rtmp2rtc.conf`

duiniuluantanqin Feb 17, 2025 •

edited

Loading

chundonglinlin commented Feb 21, 2025 •

edited by winlinvip

Loading

chundonglinlin Feb 21, 2025 •

edited by winlinvip

Loading