Good code practices

[idMysteries]

I am a new member of this project and I know python at a low level.
Let's think together about a good solution to some problems? @dipu-bd @SirGryphin

For example, a beautiful and fast code for adding volumes and chapters.

        for a in soup.select('.wp-block-column a'):
            chap_id = len(self.chapters) + 1
            vol_id = 1 + len(self.chapters) // 100
            if chap_id % 100 == 1:
                self.volumes.append({"id": vol_id})
            # end if
            self.chapters.append({
                'id': chap_id,
                'volume': vol_id,
                'url':  self.absolute_url(a['href']),
                'title': a.text.strip(),
            })
        # end for

Is it possible to improve this part and take it out of the loop?

vol_id = 1 + len(self.chapters) // 100
if chap_id % 100 == 1:
    self.volumes.append({"id": vol_id})
# end if

[idMysteries]

We need good code examples to get images, authors, titles.

It seems to me that such code could be implemented better.

        cover_img = soup.select_one('.entry-content .wp-block-image img')      
        if cover_img:
            self.novel_cover = self.absolute_url(cover_img['data-ezsrc'])
        # end if

        first_p = soup.select_one("div.entry-content p")

        for text in first_p.get_text(separator='\n', strip=True).split('\n'):
            if text.startswith('Author'):
                self.novel_author = text.replace("Author: ", "")
                break
            # end if
        # end for

[dipu-bd]

soup.select_one might return None. so it is good practice to check before using it. We do not want errors for just novel cover or author text which are not important. But we can throw errors if titles or chapter list are not found.

[SirGryphin]

@idMysteries It's more @dipu-bd than me, I'm low level if lower with python, I kind of understand most code but I mostly a copy and paste coders or I like to think I can modify code to make it do what I want most of the time but I don't write from scratch, most of mine is google and then edit to make it work lol.

If @dipu-bd decided this is better for future sources files I'm happy to use whatever works for each site.

[idMysteries]

A few years ago I studied functional programming. Perhaps adding "functionality" can improve the crawler code.
But now I don't remember much of what I studied. What a fool I am T_T

[dipu-bd]

Functional Programming is not the best practice at all times. OOP can help developers to avoid some easy mistakes. Python actually allows both types. It is a funny language

[idMysteries]

c1dd85f

It seems that this code is better... Or I'm just silly.

[dipu-bd]

this code does not test if first_p is not None type. it would halt the program with error if div.entry-content p is not found

[idMysteries]

[idMysteries]

In .NET there is such a thing as ?.
for ex.:

text = t?.get_text()

[dipu-bd]

I started this project a long time ago. I was just beginning to learn python at that time. Many of the sources here might not be the best example of good coding practice.

If I get a chance to define what is a good coding practice today, I would use decorations and typings heavily and would not use those # end if, # end def etc comments.

Anyway, just follow the common python coding practices and linting suggestions from flake8 and autopep8.

[idMysteries]

Which sites are the most popular in this project? Maybe it will be possible to find a good solution for them. And based on the new code, redo all the old ones. For example, add checks for the presence of images, write a good code example to add chapters and volumes.

Many sites are similar and if we write good code for several, we can distribute it to other sites.

[idMysteries]

It will be useful not only for me, but also for @SirGryphin and other contributors.

[idMysteries]

Now we are copying bad code, in my opinion, and changing it to work on other sites. And sometimes errors appear for example #1456

[dipu-bd]

Check the README.md and find novels with lowest version. They are the most recent one. Here are a few examples:

• https://github.com/dipu-bd/lightnovel-crawler/blob/master/sources/multi/mtlnovel.py
• https://github.com/dipu-bd/lightnovel-crawler/blob/master/sources/en/h/hostednovel.py
• https://github.com/dipu-bd/lightnovel-crawler/blob/master/sources/en/l/lightnovelreader.py
• https://github.com/dipu-bd/lightnovel-crawler/blob/master/sources/en/r/ranobes.py

[idMysteries]

I think I found a good solution to parse such lines.
next_sibling works well for this

novelcontent = soup.select_one(".novelcontent")
if novelcontent:
    author_strong = novelcontent.find('strong', string='Author: ')
    if author_strong and author_strong.next_sibling:
        self.novel_author = author_strong.next_sibling.text

[idMysteries]

Many sites use this method of specifying the author or other information. When in one div or p the "Author: " is specified in the tag (strong ex.), and then comes the author's name.

[idMysteries]

@dipu-bd fed67ab
Are we switching to double quotes instead of single quotes?

[idMysteries]

@dipu-bd holy... you've done a really great job.
Now also don't need to specify volumes?

[idMysteries]

I look at your new code and I can't stop being surprised.

[idMysteries]

Why are the templates not in the sources folder? @dipu-bd

[dipu-bd]

I am using blake for code formatting. It replaces single quote with double by default

[dipu-bd]

Why are the templates not in the sources folder? @dipu-bd

This is temporary.

[idMysteries]

@dipu-bd can you please tell us about some changes in the code? What kind of models have appeared? For example, SearchResult. How to use it?

I don't keep up with the changes, but I like them. If you have time, describe a little the changes that have occurred in recent days.

[idMysteries]

for a in soup.select('a'):
            chap_id = len(self.chapters) + 1
            self.chapters.append(Cahpter(
                'id': chap_id,
                'url':  self.absolute_url(a['href']),
                'title': a.text.strip(),
            ))

?

[idMysteries]

I also plan to replace all existing sources with templates if I have the time or patience

I CAN HELP YOU! I've been planning this from the very beginning. If you remember, I suggested an idea with templates. I liked this idea.

[idMysteries]

Just show an example of a good template with new models (SearchResults etc) on the example of Madara or new template...

[idMysteries]

I've been learning python for the last month and your code helps. I write code well only in C#, Java and a little Rust/C++. But now I've fallen in love with python 😄

[dipu-bd]

Just show an example of a good template with new models (SearchResults etc) on the example of Madara or new template...

I am currently working on it.

[idMysteries]

Some chapters have a number in the title, and some don't.
Is it possible to make it so that there is always a chapter number?

Like "Chapter X: Title", but not "Chapter X: Chapter X: Title"?

OR Maybe "Title (ch. X)", "Chapter X: Title (ch. X)"?

[idMysteries]

Or just "(X) Title", "(X) Chapter X: Title"?

@dipu-bd do you like it? For ex.:
"(104) Chapter 89.4", "(104) Chapter 89.4: World War III"?

[idMysteries]

@dipu-bd Do we need to create a separate module for each site from the template?
Can we, like here, for example, do all classes with a template class.
https://github.com/KotatsuApp/kotatsu-parsers/blob/dac06ddd0ba587f5c6333a4610acfde67f16ff33/src/main/kotlin/org/koitharu/kotatsu/parsers/site/madara/MadaraParser.kt

class MadaraTemplate(Crawler)...
...

class MangaRockteam(MadaraTemplate):
...

class Madara2(MadaraTemplate):...

class Madara4(MadaraTemplate):...

class Madara5(MadaraTemplate):...

class Madara6(MadaraTemplate):...

[idMysteries]

Dozens of modules with a ten-line code.

[dipu-bd]

You can inherit another template in your template. I did not test it. But it should work

[dipu-bd]

I prefer keeping separate file for each source. It is easier for versioning and management