All Questions

Filter by
Sorted by
Tagged with
0votes
0answers
21views

TypeError: 'dict' object is not callable in BeautifulSoup [closed]

I tried writing code that would get all the titles and links on Google search result, then print all those titles and links. from bs4 import BeautifulSoup import requests import colorama from colorama ...
user avatar
0votes
1answer
21views

bs4 `next_sibling` VS `find_next_sibling`

I struggling with usage of next_sibling (and similarly with next_element). If used as attributes I don't get anything back but if used as find_next_sibling (or find_next) then it works. From the doc: ...
user avatar
  • 1,561
1vote
1answer
19views

How to display a certain number of links along with information

How to display a certain number of links along with information? First code asks me which page to open and how many items to display. But no matter what I do, my links are displayed correctly, but the ...
user avatar
  • 47
0votes
1answer
27views

Trouble getting links from website scraper with BS4

This is the first time I'm making a website scraper and I'm relatively new to programming in general. So I'm trying to get the HREF links for all the subpages on this site:. But when I ran the code, ...
user avatar
  • 1
-3votes
1answer
24views

Beautifulsoup scraping Tripadvisor does not work [closed]

im a beginner with python and beautifulsoup for web scraping and i had an issue scraping Tripadvisor site for reviews like the code is not running it stays forever with no results . yet my code is ...
user avatar
0votes
1answer
23views

Parsing and modifying content with Beautiful Soup (bs4)

Goal is to modify existing html's content only. For example, given current markup: <html lang="en" op="item"> <head> <meta name="referrer" content="origin"> <title&...
user avatar
1vote
1answer
29views

Parsing invalid HTML and retrieving tag´s text to replace it

I need to iterate invalid HTML and obtain a text value from all tags to change it. from bs4 import BeautifulSoup html_doc = """ <div class="oxy-toggle toggle-7042 toggle-7042-...
user avatar
-1votes
1answer
53views

HTML problem with tags and classes in a simple and little scraping with BeautifulSoup

I am new and am trying to get BeautifulSoup to work. I have Html problems with recovering classes and tags. I get closer, but there is something I'm wrong. I insert wrong tags and classes to scrape ...
user avatar
  • 99
1vote
4answers
43views

Scrape a line of text from a website inside a div

I don't know how to scrape this text Telefon Mobil Apple iPhone 13, Super Retina XDR OLED 6.1", 256GB Flash, Camera Duala 12 + 12 MP, Wi-Fi, 5G, iOS (Negru) <div class="npi_name"&...
user avatar
-1votes
1answer
29views

Find Tags that Match Specific Classes but one class keeps changing

I want to extract information from a div tag which has some specific classes. Class are in the format of abc def jss238 xyz Now, the jss class number keeps changing, so after some time ,the classes ...
user avatar
-5votes
1answer
39views

Python amazon price tracker [closed]

so i wrote this code but it is aoutputing empty [] and the id productTitle exists does someone know how i fix that? import requests from bs4 import BeautifulSoup url = input("Bitte gib den ...
user avatar
  • 1
-1votes
0answers
21views

Web-scrapping problem, cant scrap sections from div - Python [closed]

Hello everyone I have a problem, I would like to extract my bets, stakes, winnings, etc. from my bookmaker, make a sheet and visualize it, calculate the profit, etc., a kind of scrap out of boredom ...
user avatar
0votes
1answer
40views

How to get specific text hyperlinks in the home webpage by BeautifulSoup?

I want to search all hyperlink that its text name includes "article" in https://www.geeksforgeeks.org/ for example, on the bottom of this webpage Write an Article Improve an Article I want ...
user avatar
  • 1,325
-1votes
0answers
17views

Python - How to decode this string that Beautifulsoup process it?

I have an HTML document that has javascript in it, using re.findall I was able to get the arguments of the function I would need to convert them to a Beautifulsoup object. The problem is that BS can ...
user avatar
  • 39
-1votes
2answers
44views

webscraping from cnn function to get text from a article error in python

So i want to get the text from a specific article(not only one) so heres the function therefore: def get_article(): for url in get_href(): options = webdriver.ChromeOptions() ...
user avatar
  • 1
0votes
0answers
39views

Python Beautifulsoup: < and > become &lt; and &gt;

I am doing some web scraping using beautifulsoup4. However, half of the website gets scrambled by < and > being replaced with &lt; and &gt;. What am I doing wrong? Why does beautifulsoup ...
user avatar
0votes
1answer
32views

Why it is showing Index error and Incomplete data scrape while doing python web scraping?

I am trying to scrape the web page "https://global.oup.com/academic/content/series/v/very-short-introductions-vsi/?type=listing&lang=en&cc=in" after I run the script, it gives the ...
user avatar
-1votes
0answers
26views

Web Scraping: Frequently Broken with Error 503 Service Unavailable

I am scraping a website with like 20,000+ items. During the scraping, I frequently get broken. The codes are fine, as I can continue to scrap the website by manually continuing the codes. For example, ...
user avatar
0votes
1answer
38views

how do i get data from function, var in scripts using python?

<script defer=""> window.__CURRENT_SITE__ = window.__CURRENT_SITE__ || "videoblocks"; window.__CURRENT_PATH__ = window.__CURRENT_PATH__ || "\/video\/stock\/...
user avatar
-1votes
1answer
44views

Error when webscraping news from cnn using selenium and bs4 to get links and titles from articles

I wrote this code for now to webscrape news from a spacific topic from cnn: from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.chrome.service import Service ...
user avatar
-1votes
1answer
34views

cnn news webscraper return empty [] without information

so i wrote this code for now: from urllib import request from bs4 import BeautifulSoup import requests import csv import re serch_term = input('What News are you looking for today? ') url = f'https:/...
user avatar
-1votes
1answer
21views

Python beautifulsoup find_all can‘t find <div class=“ ”>

I'm trying to use beautifulsoup to find content in HTML tags. But when the tags are /div class=" "/ , it doesn't work. It cannot be recognized correctly when there is a space in double ...
user avatar
  • 1
1vote
1answer
35views

requests.exceptions.InvalidURL: Failed to parse: <Response [200]> in python

So i wrote this code for now, to get news from a specific topic from cnn right now im getting an error here is the code: from bs4 import BeautifulSoup import requests import csv import re serch_term =...
user avatar
-2votes
0answers
17views

find_all() function in web scraping not getting full result

import requests from bs4 import BeautifulSoup url = 'https://www.espncricinfo.com/series/ipl-2020-21-1210595/delhi-capitals-vs-mumbai-indians-final-1237181/ball-by-ball-commentary' response = requests....
user avatar
  • 1
0votes
1answer
26views

How to make our webscraping script check both scenarios but execute only the one needed

I scrape some data on website, here's my script : import warnings warnings.filterwarnings("ignore") import re import requests from requests import get from bs4 import BeautifulSoup import ...
user avatar
-1votes
1answer
46views

capture updated exchange rate by using python

I am trying to capture exchange rate from hexun.com, from bs4 import BeautifulSoup import requests urls = 'http://so.hexun.com/default.do?type=forex&key=iskcny' html = requests.get(urls) soup = bs(...
user avatar
0votes
1answer
25views

My code cannot append next pages tables to the end of the list

I trying to scrap all tables of 8 pages but my code just scrap 1st table. It can move to other pages also it works individually on each page but it cannot scrap all pages. data_ingram = [] n = 1 for i ...
user avatar
  • 63
0votes
0answers
18views

How to scrape the data-block-id attribute in a dymanic webpage? [duplicate]

I'm scraping some information of this homepgae (www.globo.com). I would like to scrape the attribute "data-block-id", which is located inside the "a" tag (that contains the URL of ...
user avatar
  • 173
0votes
1answer
56views

I've issues with getting data with BeautifulSoup

import requests from bs4 import BeautifulSoup URL = "https://www.empireonline.com/movies/features/best-movies-2/" response = requests.get(URL) website_html = response.text soup = ...
user avatar
1vote
3answers
46views

Click a date range button and crawler one html table in Python

I try to crawler a small table data from here, the process is shown by the figure below: import requests from bs4 import BeautifulSoup import pandas as pd url = 'https://oilprice.com/rig-count' # ...
user avatar
  • 7,539
0votes
1answer
29views

Extract key in messy website with Beautiful soup

I'm new in webscraping with beautiful soup and I have some problems... Here is my code from bs4 import BeautifulSoup import numpy as np from time import sleep from random import randint from selenium ...
user avatar
0votes
1answer
20views

Scraping points on a graph on a website using beautifulsoup or selenium

I want to get the values of the data points from the graph titled "Total Followers for 'OlympusDAO' (Monthly)" from this website: https://socialblade.com/twitter/user/olympusdao/monthly Here'...
user avatar
  • 143
-1votes
0answers
28views

I'm trying to web scrape a website that's behind a login page with Python, but it's telling me I'm not logged in

So, I'm trying to log into a website using Python. I've looked at a ton of different tutorials, and have followed what they've said, but it isn't working. This is what the payload on the website looks ...
user avatar
0votes
2answers
30views

scraping table using beautiful soup

I'm trying to scrape the address/amount/share columns of the top 100 holders on this website: https://cryptorank.io/price/butterflydao/holders This is my code: import requests from bs4 import ...
user avatar
  • 143
0votes
0answers
32views

HTML file size changes after reading and writing to a new file using beautifulsoup

I have an HTML file here of File size - 1128 KB. Following shows the size of the above HTML file: Following is my code to just read and write the HTML file to a new file sample.htm: from bs4 import ...
user avatar
  • 13
-1votes
1answer
42views

ValueError: Cannot convert <.....><....> to Excel

hi im new to python programming. im try to web scraping a news website using python. I got the title and its links. But when i try to save it in excel file it shows value error Here is Source code and ...
user avatar
1vote
2answers
27views

Trying to scrape text from a site with BeautifulSoup4, but nothing happens at all

I want to scrape data from this website: https://playvalorant.com/en-us/news/game-updates/ from bs4 import BeautifulSoup import requests site_text = requests.get('https://playvalorant.com/en-us/news/...
user avatar
  • 27
-2votes
1answer
58views

how to make this code run many times based on "page_num" varibale, to scrape all pages? using BeautifulSoup

I'm trying to scrape the websitehttps://www.bayut.sa/en/riyadh-region/villas-for-sale-in-riyadh/page-2/, the code succeeded to scrape the first page only which is page-2 here, but it does not work to ...
user avatar
  • 1
1vote
3answers
38views

Preserve &nbsp; in beautiful soup object

I have my sample.htm file as follows: <html><head> <title>hello</title> </head> <body> <p>&nbsp; Hello! he said. &nbsp; !</p> </body> </...
user avatar
  • 13
-1votes
0answers
22views

Web Scraping using Beautiful Soup- Change filters on website and download data

I am using the below given code to download excel file from the URL- https://www.iexindia.com/marketdata/market_snapshot.aspx The data being downloaded is for 'Delivery Period' filter 'Today'. I would ...
user avatar
0votes
1answer
22views

Trouble with scraping links with BeautifulSoup

Here's my script : import requests from bs4 import BeautifulSoup import pandas as pd headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0', ...
user avatar
-1votes
0answers
14views

just add the last page elements in list

I'm trying to scrape tables info that don't have ID and Class from 8 pages and add in a DataFrame. but my code just add the last page elements in list. pagenum = 10 data_ingram = [] i = 1 for i in ...
user avatar
  • 63
1vote
1answer
17views

BeautifulSoup - Is there a way to find starting from a specific row number?

I am using python and BeautifulSoup for making a discord bot I have my code: URL = "https://www.mywebsite.com" with requests.Session() as s: r = s.post(URL) soup = ...
user avatar
-1votes
1answer
50views

Splitting problem: 'NoneType' object is not callable

I have been facing a small problem with splitting/slicing string : import requests from bs4 import BeautifulSoup url = 'http://www.example.com' r = requests.get(url) soup = BeautifulSoup(r.text, '...
user avatar
  • 11
0votes
0answers
8views

Beautiful Soup not returning all the tags [duplicate]

I tried running the below program, but it only returns first 9 entries from the webpage but I need all the listings shown in the url which is approx 40 import requests from bs4 import BeautifulSoup ...
user avatar
-2votes
2answers
43views

how to get the base string and page no string in for loop?

currently i am putting the full url in urlist i want the only string after pageno in the urlist and the program should go on rest as it as. https://bidplus.gem.gov.in/bidlists?bidlists&page_no=**...
user avatar
2votes
0answers
37views

how to navigate to multiple pages Webscrap by beautiful soup when page no is encrypted?

i used to web scrape a site which cantains 1000 pages and i wused to traverse each page with page no as 1,2,3...1000 and download data in excel now they have encrypted the page no. so code is no ...
user avatar
1vote
1answer
48views

How can I find an element by screen location with Python Selenium?

I've been searching online but haven't found any answers yet on how to select an element by screen position with selenium. I've found ways to get the position of an element once you've selected it but ...
user avatar
  • 11
-1votes
1answer
42views

How to extract the text when scraping p tags and br tags

I have an issue with scrapping using Beautiful Soup. I want the text from: url_example = 'https://www.verychic.fr/p/21/brunelleschi-hotel-s' Which should be: <p data-v-7816a06c="" class=&...
user avatar

15 30 50 per page
1
2 3 4 5
96