0

I am wondering how I could scrape data from a website that requires my login, and login automatically using cookies so I won't get blocked by verification when I try to login with a username and a password.

For example, I have a channel in discord that can only be viewed if I am logged in on my account.

url = https://discord.com/channels/742886974288822413/743510591670583306

Is there any way I could send a request to this website and put my cookies in as an argument so I am logged in by default when I am scraping this site? I despise using selenium as it is very slow, and opens up an entire browser on my system. I would prefer using a tool that doesn't open up an entire browser, such as the requests module or urllib.

John Kugelman
  • 349,597
  • 67
  • 533
  • 578
Pro Dude
  • 33
  • 5
  • 1
    Does this answer your question? [How to use cookies in Python Requests](https://stackoverflow.com/questions/31554771/how-to-use-cookies-in-python-requests) – biomiker Nov 20 '20 at 19:43
  • Sort of but I am confused what it means by payload what should that variable be equal to? – Pro Dude Nov 20 '20 at 19:52
  • @biomiker would the login be to my discord account or google account when it says: `payload={'username'=,'password'=}`? – Pro Dude Nov 20 '20 at 19:58
  • @biomiker The payload above is invalid syntax on my pycharm I think that answer is outdated. – Pro Dude Nov 20 '20 at 20:03
  • I am so confused nothing makes sense what code what I use to scrape the data from that website, and how would I send the request using the cookies from my default chrome browser? – Pro Dude Nov 20 '20 at 20:06
  • Sorry, payload is the content that is being posted in that example. You only need to do a GET operation so you wouldn't need to submit a payload. However, I assumed from your question that you can submit your credentials using only a cookie. Do you know if that's the case or not? – biomiker Nov 20 '20 at 20:24
  • I am not sure which credentials to post. – Pro Dude Nov 20 '20 at 21:00
  • So which code would I execute and where would I type my credentials? – Pro Dude Nov 20 '20 at 21:01
  • That's a different question. It depends on the manner in which the website authenticates. However, I think you should abandon the idea of trying to scrape the web interface and look into discord's developer portal instead. https://discord.com/developers/docs/intro – biomiker Nov 21 '20 at 19:54

0 Answers0