Selenium gets stuck in Login phase, even the real browser is already logged in (needs session data)

Question

Unlike the usual, I am having trouble while fetching elements from the website below.

The element hierarchy is as follows:

My goal is to fetch the rows (data-row-key='n') inside the class "rc-table-tbody".

Here below is my Python script:

chromeOptions = Options()
chromeOptions.add_argument("headless")
driver = webdriver.Chrome(chrome_options=chromeOptions, executable_path=DRIVER_PATH)
driver.get('https://www.binance.com/en/my/orders/exchange/usertrade')

None of these below works (gives either unable to locate element or timeout):

elements = driver.find_element_by_class_name("rc-table-tbody")
elements = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, "rc-table-tbody")))
elements = WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "rc-table-tbody")))
elements = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "selector_copied_from_inspect")))
elements = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "xpath_copied_from_inspect")))

I appreciate any help, thanks!

Edit: I guess the problem is related to cookies. The URL I was trying to fetch is a page after a login in Binance.com and I am already logged in on my Crome browser. Therefore, I assumed that driver will use the current cookies of the real Chrome browser and there wouldn't need for login. However, when I removed "headless" parameter, Selenium poped the login page, instead of the page I was trying to scrape.

How can I get the current cookies of the browser, in order to make Selenium access the exact page I am trying to scrape?

Check the DOM tree..if the target element present inside an `iframe`? — KunduK, Jan 14 '21 at 16:17
You need to create an [MRE](https://stackoverflow.com/help/minimal-reproducible-example). It's almost impossible to help you without knowing what URL you are going against. Otherwise, you just get guesses, one of which might be right -- or not. — Booboo, Jan 14 '21 at 16:17
@KunduK I cannot see any iframe tag, as you can see in the first image. My target is the highlighted one (rc-table-tbody) and the elements under that. — bbasaran, Jan 14 '21 at 17:09
@Timeler your suggestion led me to see the problem, thanks! I edited the question, can you please check? — bbasaran, Jan 14 '21 at 17:19
@Booboo You are right, I updated the URL, and I have discovered the problem. I edited the question, could you please check it? — bbasaran, Jan 14 '21 at 17:20
@bbasaran Login credentials are being requested at that URL. — Booboo, Jan 14 '21 at 17:26
@Booboo yes, but I am already logged in in the real Chrome browser. How can I make Selenium see that session data while opening the URL? — bbasaran, Jan 14 '21 at 17:29
you could create a new firefox profile with the respective cookies — Timeler, Jan 14 '21 at 17:56

score 0 · Answer 1 · answered Jan 14 '21 at 18:14

You would have to do some research but I believe you can set up a special Chrome profile for use with "remote debugging" and then start up chrome with remote debugging. See Remote debugging with Chrome Developer Tools. Then there is a way to start up your selenium chromedriver with the correct options (usually something like chromeOptions.add_argument('--remote-debugging-port=9222')) so as to hook into this special Chrome where you have already logged on.

But it might be far simpler to insert an input("Pausing for login completion...") statement following the driver.get call, then login manually and when you have finished successfully logging in, hit the enter key in reply to the input statement to resume execution of your program. For this option you could not use "headless" mode. Of course, this latter option is not a "hands-off", long-term solution. But I looked at the possibility of just adding logic to your existing code to send login credentials, but there seems to be on the next page a nasty captcha that would be very difficult to get past.

Thank you for your valuable comments @Booboo Starting from your second suggestion, I would say that this is not much an option for my case because I am trying to write a Chrome extension for a quick calculation. User shouldn't have to deal with login. I have tried the following for fetching the session data, I don't know why but it failed: options.add_argument("--user-data-dir=/home/bbasaran/snap/chromium/common/chromium/Default") — bbasaran, Jan 14 '21 at 23:01
There is a relevant post [here](https://stackoverflow.com/questions/31062789/how-to-load-default-profile-in-chrome-using-python-selenium-webdriver), but comments suggest that this does not work. You should look into the first suggestion. You just need to start up Chrome with special debugging options prior to running the selenium code and then selenium should connect to that existing running Chrome session. Anyway, that is the theory. — Booboo, Jan 14 '21 at 23:29

Selenium gets stuck in Login phase, even the real browser is already logged in (needs session data)

1 Answers1