32,797
questions
52
votes
4
answers
115k
views
Extract content within a tag with BeautifulSoup
I'd like to extract the content Hello world. Please note that there are multiples <table> and similar <td colspan="2"> on the page as well:
<table border="0" cellspacing="2" width="800"...
- 41.6k
1
vote
2
answers
4k
views
403 Forbidden Error when scraping a site, user-agents already used and updated. Any ideas?
As the title above states I am getting a 403 error. The URLs generated are valid, I can print them and then open them in my browser just fine.
I've got a user agent, it's the exact same one that my ...
- 28.4k
-1
votes
1
answer
1k
views
Web Scraping in R / Python
I need to extract data from https://eservices.dha.gov.ae/DHASearch/UIPages/ProfessionalSearch.aspx?PageLang=En. I need 4 columns -"name","gender", "Titles" ,"Hospital Name", "Contact details". The "...
CommunityBot
- 1
-2
votes
0
answers
20
views
Beautifulsoup not finding "li" class when a url with search word is given [closed]
I am trying to extract "li" class variable using BeautifulSoup from a ULTA.com for cosmetic products.
Code:
page = requests.get(url)
soup = BeautifulSoup(page.content,'html.parser')
...
- 1
0
votes
0
answers
24
views
Where do I look to find a domain's policy on web scraping? [closed]
I often use bs4 and requests in python to gather data from a website.
Where can I find the policy on data collection? (Keywords, Sections, shortcuts, etc.)
I'm hesitant to post any of my code on the ...
- 45
2
votes
1
answer
43
views
Get an <a> tag content using BeautifulSoup
I'd like to get the content of an <a> tag using BeautifulSoup (version 4.12.3) in Python.
I have this code and HTML exemple:
h = """
<a id="0">
<table>
...
- 194k
-1
votes
1
answer
77
views
How can I export text from a specific div with class "swatch-option text" using Python and BeautifulSoup?
I'm trying to scrape shoe sizes from a website using Python and BeautifulSoup. The shoe sizes are located in a div with the class "swatch-option text." I've already managed to extract other ...
- 28.4k
-4
votes
0
answers
32
views
Efficient Strategies for Scraping Multiple Pages from a Website [closed]
I am developing a web scraper to extract data (such as titles, authors, and prices) from a website that has potentially up to 2100 pages. I’m seeking advice on the most efficient strategy for handling ...
0
votes
1
answer
36
views
Can't get all span tags inside div element using beautifulsoup
I am scraping the product details page text on Amazon, but I get the data back as bullet list. I want to have the data added as a column next to other scraped data.
csv export
Amazon Product Details
A ...
- 194k
0
votes
0
answers
26
views
Sending Chunks to Mistral API and Handling Streaming Responses
I need to send HTML content in chunks to the Mistral API and receive the responses for all chunks in one go. My code processes the content in chunks and sends them to the Mistral API using streaming. ...
- 59.9k
0
votes
2
answers
84
views
scrape a website which has the same url for multiple pages? with the page jump being an ajax request
I've been at this for days, I'm trying to scrape this website: "https://careers.ispor.org/jobseeker/search/results/".
I've got everything covered, from the script that will extract the ...
- 1,521
0
votes
2
answers
561
views
Web scraping table from UniProt database
I have a list of UniProt IDs and would like to use BeautifulSoup to scrap a table containing the structure information. The url I am using is as follows: https://www.uniprot.org/uniprot/P03496, with ...
- 9,058
3
votes
1
answer
4k
views
Headless doesn't work using Playwright and BeautifulSoup 4
This code is working:
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
from datetime import datetime
import time
with sync_playwright() as p:
browser = p.chromium....
- 55k
3
votes
3
answers
917
views
How do I extract tags with ::marker from HTML using beautiful soup
I am trying to find li elements that have ::marker such as seen below using BeautifulSoup.
I tried using cssutils but was unsuccessful (maybe I am using it wrong)
Pseudo code:
lis = soup_obj.find_all(...
- 147
1
vote
1
answer
989
views
Not able to scrape the all the reviews
I am trying to scrape this website and trying to get the reviews but I am facing an issue,
The page loads only 50 reviews.
To load more you have to click "Show More Reviews" and I don't ...
- 21.1k