New feature web parser with pattern #950

alpgul · 2025-01-31T18:22:09Z

Description

Implement WebStreamExtractor to support dynamic URL extraction from web sources in playlist loading. This includes:

New WebStreamExtractor utility class for extracting stream URLs
Updated PlaylistLoader to support '@' prefixed URLs with optional web pattern
Added CMakeLists.txt to include new source files
Added Windows build script for easier compilation

Missing Features

Missing header support for web requests(CFile.OpenFile function doesn't support web header. what alternatives can be used?)
In-depth web crawling is currently not supported.

Example Usage

usage with default pattern:

#EXTINF:-1 tvg-name="Test" , Test
@http://127.0.0.1:3000/index.html

usage with custom pattern:

#EXTINF:-1 tvg-name="Test1", Test1
#WEBPROP:web-regex="(https?://[^\"]+\.m3u8)"
@http://127.0.0.1:3000/index.html

URL must be specified in parentheses.

phunkyfish · 2025-01-31T21:39:46Z

Could you provide some examples? Preferably real world use cases.

The purpose of the feature is not clear to me from the PR description.

alpgul · 2025-01-31T22:27:54Z

It is used to search for media URLs in HTML pages using a regex pattern.
Example usage: Define an m3u8 link in an HTML page. Then, define a regex to find the media URL link, and add the html link and regex as shown in the example usage above.
Example Html Page:

<html>
...
url:"https://localhost:3000/index.m3u8"
...
</html>

Example Regex:#WEBREGEX:url:"(https?://[^"]+\.m3u8)"

The current issue is that I can't send requests using custom headers. Which library should I use that works on all platforms? I'm considering using CURL, but does it work on all platforms?

alpgul · 2025-02-01T14:46:38Z

Added support for headers in web URL requests.

Example Usage

#EXTM3U
#EXTINF:-1 tvg-name="Test1", Test1
#WEBPROP:web-regex="([^"]+\.mp4)"
#WEBPROP:web-headers=user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36&referer:https://google.com/
@https://test-videos.co.uk/bigbuckbunny/mp4-h264

phunkyfish · 2025-02-07T08:17:51Z

How do you ensure only one URL is selected, and also ensure it is the correct one?

phunkyfish · 2025-02-07T08:18:26Z

Also note that new features must be tested and merged on the Piers branch before being considered for the Omega branch.

phunkyfish

Just a light first review. I'd suggest collapsing all commits into a single one, as many of the commits change the previous one, making it difficult to review.

Also, many of the files are missing a newline at the end of the file.

build-windows.bat

src/iptvsimple/PlaylistLoader.cpp

src/iptvsimple/utilities/WebStreamExtractor.cpp

src/iptvsimple/utilities/WebStreamExtractor.h

src/iptvsimple/PlaylistLoader.cpp

src/iptvsimple/utilities/WebStreamExtractor.cpp

src/iptvsimple/utilities/STRING.cpp

phunkyfish · 2025-02-07T08:45:28Z

build-windows.bat should be a separate commit.

alpgul · 2025-02-07T12:38:16Z

How do you ensure only one URL is selected, and also ensure it is the correct one?

Initially, the URL fetching process was in the playlist loader. However, now that it has been moved to "GetChannelStreamProperties" there should be no issues.

build-windows.bat should be a separate commit.

It has been moved to a separate commit.

I merged the all commits.

phunkyfish

Ok, review round 2.

Please take note https://github.com/xbmc/xbmc/blob/master/docs/CODE_GUIDELINES.md

depends/common/zlib/01-build-static.patch

depends/common/zlib/03-install-pkgconfig-in-lib.patch

depends/common/zlib/zlib.sha256

src/IptvSimple.cpp

src/iptvsimple/PlaylistLoader.cpp

phunkyfish · 2025-02-08T07:55:58Z

src/iptvsimple/utilities/Base64.cpp

+#include "Base64.h"
+
+using namespace iptvsimple;
+using namespace utilities;
+namespace
+{
+constexpr char PADDING{'='};
+constexpr std::string_view CHARACTERS{"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+                                      "abcdefghijklmnopqrstuvwxyz"
+                                      "0123456789+/"};
+// clang-format off
+constexpr unsigned char BASE64_TABLE[] = {


What about using https://github.com/azawadzki/base-n instead as a dependency? It support base64 out of the box. Then we don't need to carry this extra code.

As far as I can tell it's standard C++ so should be cross platform.

I pulled this code from inputstream.adaptive. I chose this to minimize the risk of errors. If base-n is more performant, can we first adapt the changes in inputstream.adaptive and then apply them here as well?
Reference Code Link

I would probably take a different view on this. We should try the non custom code approach in iptvsimple and if that proves to work correctly then propose that dependency change for adaptive but not the other way around.

src/iptvsimple/utilities/CurlUtils.cpp

alpgul · 2025-02-08T15:01:40Z

Ok, review round 2.

Please take note https://github.com/xbmc/xbmc/blob/master/docs/CODE_GUIDELINES.md

https://github.com/xbmc/xbmc/blob/master/.clang-format

The .clang-format file is not working properly because when I apply automatic formatting, the content of the code changes completely. In other words, the formatting file is incompatible with the previous code style, and I have to manually fix the code every time. If you have one, could you share the .clang-format file you are using?

alpgul · 2025-02-08T15:16:29Z

src/iptvsimple/utilities/WebStreamExtractor.cpp

+
+using namespace iptvsimple::utilities;
+
+std::string WebStreamExtractor::ExtractStreamUrl(const std::string& webUrl,


This code explained in this link

src/iptvsimple/utilities/CurlUtils.cpp

phunkyfish · 2025-02-10T08:35:25Z

Ok, review round 2.
Please take note https://github.com/xbmc/xbmc/blob/master/docs/CODE_GUIDELINES.md

https://github.com/xbmc/xbmc/blob/master/.clang-format

The .clang-format file is not working properly because when I apply automatic formatting, the content of the code changes completely. In other words, the formatting file is incompatible with the previous code style, and I have to manually fix the code every time. If you have one, could you share the .clang-format file you are using?

That clang format file is for xbmc, we don't follow all the conventions in addons just some of the more prominent ones to make code easier to read. I understand that doesn't make it black and white for contributors but it's where we are currently.

src/iptvsimple/utilities/CurlUtils.cpp

alpgul · 2025-02-11T07:54:10Z

Removed Unused Methods in CurlUtils and Base64

phunkyfish requested changes Feb 7, 2025

View reviewed changes

alpgul force-pushed the newFeature-webParserWithPattern branch from 9d67352 to 6603db2 Compare February 7, 2025 11:57

Add Windows build script for PVR IPTV Simple addon

7f3c18c

alpgul force-pushed the newFeature-webParserWithPattern branch from 6603db2 to 08a4ca9 Compare February 7, 2025 12:31

phunkyfish requested changes Feb 8, 2025

View reviewed changes

Add web stream extraction support for IPTV Simple addon

a0627e9

alpgul mentioned this pull request Feb 8, 2025

New Feature: Web parser with pattern for Piers #957

Open

alpgul force-pushed the newFeature-webParserWithPattern branch from 08a4ca9 to a0627e9 Compare February 8, 2025 14:48

alpgul commented Feb 8, 2025

View reviewed changes

src/iptvsimple/utilities/CurlUtils.cpp Show resolved Hide resolved

Improve web stream URL extraction handling

40d3de1

phunkyfish requested changes Feb 10, 2025

View reviewed changes

src/iptvsimple/utilities/CurlUtils.cpp Show resolved Hide resolved

Remove Base64 utility and simplify CurlUtils

b9fa3c2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New feature web parser with pattern #950

New feature web parser with pattern #950

alpgul commented Jan 31, 2025 •

edited

Loading

phunkyfish commented Jan 31, 2025 •

edited

Loading

alpgul commented Jan 31, 2025

alpgul commented Feb 1, 2025 •

edited

Loading

phunkyfish commented Feb 7, 2025

phunkyfish commented Feb 7, 2025

phunkyfish left a comment

phunkyfish commented Feb 7, 2025

alpgul commented Feb 7, 2025

phunkyfish left a comment

phunkyfish Feb 8, 2025

alpgul Feb 8, 2025 •

edited

Loading

phunkyfish Feb 10, 2025

alpgul commented Feb 8, 2025

alpgul Feb 8, 2025 •

edited

Loading

phunkyfish commented Feb 10, 2025

alpgul commented Feb 11, 2025


		using namespace iptvsimple::utilities;

		std::string WebStreamExtractor::ExtractStreamUrl(const std::string& webUrl,

New feature web parser with pattern #950

Are you sure you want to change the base?

New feature web parser with pattern #950

Conversation

alpgul commented Jan 31, 2025 • edited Loading

Description

Missing Features

Example Usage

phunkyfish commented Jan 31, 2025 • edited Loading

alpgul commented Jan 31, 2025

alpgul commented Feb 1, 2025 • edited Loading

Example Usage

phunkyfish commented Feb 7, 2025

phunkyfish commented Feb 7, 2025

phunkyfish left a comment

Choose a reason for hiding this comment

phunkyfish commented Feb 7, 2025

alpgul commented Feb 7, 2025

phunkyfish left a comment

Choose a reason for hiding this comment

phunkyfish Feb 8, 2025

Choose a reason for hiding this comment

alpgul Feb 8, 2025 • edited Loading

Choose a reason for hiding this comment

phunkyfish Feb 10, 2025

Choose a reason for hiding this comment

alpgul commented Feb 8, 2025

alpgul Feb 8, 2025 • edited Loading

Choose a reason for hiding this comment

phunkyfish commented Feb 10, 2025

alpgul commented Feb 11, 2025

alpgul commented Jan 31, 2025 •

edited

Loading

phunkyfish commented Jan 31, 2025 •

edited

Loading

alpgul commented Feb 1, 2025 •

edited

Loading

alpgul Feb 8, 2025 •

edited

Loading

alpgul Feb 8, 2025 •

edited

Loading