All Questions
Tagged with beautifulsoup python-requests
3,044
questions
0
votes
0
answers
24
views
Where do I look to find a domain's policy on web scraping? [closed]
I often use bs4 and requests in python to gather data from a website.
Where can I find the policy on data collection? (Keywords, Sections, shortcuts, etc.)
I'm hesitant to post any of my code on the ...
- 45
-4
votes
0
answers
32
views
Efficient Strategies for Scraping Multiple Pages from a Website [closed]
I am developing a web scraper to extract data (such as titles, authors, and prices) from a website that has potentially up to 2100 pages. I’m seeking advice on the most efficient strategy for handling ...
0
votes
1
answer
28
views
HTTP Error 404 when scraping first table using BeautifulSoup, but second table works fine
I’m working on a Python script to scrape historical CDS data from Investing.com using BeautifulSoup. The goal is to extract data from a specific table on the page and compile it into a DataFrame.
Here’...
- 5
0
votes
1
answer
60
views
Optimize Python Web Scraping Script Using concurrent.futures to Reduce Execution Time
I'm currently working on a web scraping script in Python that extracts table data from multiple pages of a website using urllib, BeautifulSoup, and pandas. The script is designed to handle content ...
- 297
2
votes
1
answer
55
views
Scrape the latitude and longitude from the website
I want to convert a list of zip codes into a DataFrame of latitude and longitude using data from this website: Free Map Tools.
https://www.freemaptools.com/convert-us-zip-code-to-lat-lng.htm#...
- 23
1
vote
1
answer
45
views
Unable to produce the result containing the address I wish to obtain
I'm trying to create a script using the requests module and the BeautifulSoup library from this website that will do the following:
Select the Strata plan number button, input 11 in the input box, and ...
- 147
0
votes
2
answers
131
views
Unable to parse the results from a webpage using the requests module
I've created a script to scrape the names of the products from this webpage using the requests module. When I run the script, I can see the status code is 200, but the script doesn't bring any results....
- 53
-3
votes
1
answer
74
views
How do I fix my code, it is returning an empty list?
I am scraping an ecommerce website and its returning an empty list
This is the code I wrote.
import requests
from bs4 import BeautifulSoup
baseurl = 'https://www.thewhiskyexchange.com/'
headers = {'...
1
vote
1
answer
231
views
Response ended prematurely while scrapping web page inside cronjob
I created Cronjob to execute the Script every 24 hours, I noticed that this error occurs when the code compiles itself during the cron process on the local machine I did not notice this problem.
...
- 142
0
votes
2
answers
61
views
web scraper is not grabbing desired text
I am trying to scrape the sku and description on this site:
https://www.milwaukeetool.com/products/power-tools/drilling/drill-drivers
but, it wont scrape the desired elements despite the code being ...
0
votes
1
answer
20
views
BS4 and requests only finding and scraping first listed item in dropdown list
I have a scraper that I have written to fetch some data from 2 different URLS. The only difference between the two pages is that one has a dropdown list of menus while the other does not. The scraper ...
-1
votes
1
answer
55
views
Python: scraping links from multiple urls
I'm trying to adapt the following code to scrape links from various pages. For example, if each page has 40 links and I'm scraping 10 pages, then I expect to get 400 links total.
The web pages follow ...
-2
votes
3
answers
100
views
How do I get text from an embedded map on a website?
I have written a code that accesses a webpage and searches the page for the link to another website from the inspect element. After accessing that website, I need to get the zip code of an address ...
- 19
-1
votes
1
answer
116
views
Scrape data from website with complex structure
I am trying to scrape data from the TransferMarkt website in Python. However, the website structure is complex. I've tried using the requests and Beautiful Soup modules and the following code. However,...
0
votes
1
answer
57
views
BeautifulSoup output not properly formatted
I'm trying to webscrape some text from a website, the problem is its HTML formatting.
<div class="coptic-text html">
<div class="htmlvis"><t ...
- 1,780