Lyrics: Refactor Genius, Google backends, and consolidate common functionality #5474
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
snejus
force-pushed
the
lyrics-general-improvements
branch
5 times, most recently
from
13b52ec
to
fed32d9
Compare
snejus
force-pushed
the
fix-lrclib-lyrics
branch
from
e914d59
to
73e4a5a
Compare
snejus
force-pushed
the
lyrics-general-improvements
branch
5 times, most recently
from
1497893
to
cb559f0
Compare
snejus
force-pushed
the
fix-lrclib-lyrics
branch
from
8d6762f
to
16188fa
Compare
This commit introduces a distance threshold mechanism for the Genius and Google backends. - Create a new `SearchBackend` base class with a method `check_match` that performs checking. - Start using undocumented `dist_thresh` configuration option for good, and mention it in the docs. This controls the maximum allowable distance for matching artist and title names. These changes aim to improve the accuracy of lyrics matching, especially when there are slight variations in artist or title names, see #4791.
Improve requests performance with requests.Session which uses connection pooling for repeated requests to the same host. Additionally, this centralizes request configuration, making sure that we use the same timeout and provide beets user agent for all requests.
Having removed it I fuond that only the Genius lyrics changed: it had en extra new line. Thus I defined a function 'collapse_newlines' which now gets called for the Genius lyrics.
Tidy up 'Google.is_page_candidate' method and remove 'Google.sluggify' method which was a duplicate of 'slug'. Since 'GeniusFetchTest' only tested whether the artist name is cleaned up (the rest of the functionality is patched), remove it and move its test cases to the 'test_slug' test.
* Type the response data that Google Custom Search API return.
* Exclude some 'letras.mus.br' pages that do not contain lyric.
* Exclude results from Musixmatch as we cannot access their pages.
* Improve parsing of the URL title:
- Handle long URL titles that get truncated (end with ellipsis) for
long searches
- Remove domains starting with 'www'
- Parse the title AND the artist. Previously this would only parse the
title, and fetch lyrics even when the artist did not match.
* Remove now redundant credits cleanup and checks for valid lyrics.
Additionally, improve HTML pre-processing: * Ensure a new line between blocks of lyrics text from letras.mus.br. * Parse a missing last block of lyrics text from lacocinelle.net. * Parse a missing last block of lyrics text from paroles.net. * Fix encoding issues with AZLyrics by setting response encoding to None, allowing `requests` to handle it.
If we get caught by Cloudfare, it forwards our request somewhere else and returns some validation text response. To make sure that this text does not get assumed for lyrics, we can disable redirects for the Google backend, check the response code and raise if there's a redirect attempt. This source will then be skipped and the backend continues with the next one.
I think we can make our life easier by removing these checks assuming that users follow the instructions in the docs.
snejus
force-pushed
the
lyrics-general-improvements
branch
from
cb559f0
to
95c954a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
I've made several updates to the lyrics plugin to enhance its functionality and address a few bugs. Here's a quick overview of what has changed:
Bug Fixes
Plugin Enhancements
TimeoutSessionto enable connection pooling and maintain consistent configuration across requests.RequestsHandlerclass, which includes methods for retrieving either HTML text or JSON data.Configuration Changes
dist_threshfield to the configuration, allowing users to control the maximum tolerable mismatch between the artist and title of the lyrics search result and their item. Interestingly, this field was previously available (though undocumented) and used in theTekstowobackend. Now, this threshold has also been applied to Genius and Google search logic.Backend Updates
dist_thresh.Genius
Google
dist_threshvalue are discarded. The previous functionality simply selected the first result containing the track's title in its URL, which often led to returning lyrics for the wrong artist, particularly for short track titles.HTML Cleanup
Htmlclass.letras.mus.brandmusica.com.paroles.netandlacoccinelle.net. See the diff intest/plugins/lyrics_page.py.To Do
docs/to describe it.docs/changelog.rstat the bottom of one of the lists near the top of the document.Note: If you find this PR too large to manage, I am happy to split it up!