You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We've spent a couple weeks adding support for Bidi to Playwright and running Playwright tests with the new protocol. The good news is there has been a large progress in Bidi support in Firefox and Chromium, many Playwright tests are now passing. However, there is still a long list of issues that are blocking Playwright from adopting Bidi as the browser automation protocol. At this point the number of bugs we identified is too large to file individual reports for each of them, instead we'll use this issue to describe some of the major problems that we've discovered.
🎉 Current stats are the following:
Chromium: 2368 / 3877 passed (3.2m)
Firefox: 1469 / 3877 passed (4.1m)
Spec
Provide a way to configure the viewport before a popup or new tab starts loading content. Currently, the browsingContext.SetViewport command can only be sent to an existing page, and there is no way to pause the load of a new page (e.g., opened via ctrl+click or a popup); these pages always load with the default viewport first.
Allow to configure options(aka capabilities) such as proxy and acceptInsecureCerts per user context rather than per session, so that same browser process can be reused for testing with different options.
Add support for isMobile emulation to enable fixed layout mode.
Allow specifying a screen size that is different from the viewport size.
Include support for landscape mode emulation.
Add support for getting content quads to ensure accurate clicking inside transformed elements. This is particularly relevant for tests involving transformed elements, as seen in tests/page/page-click.spec.ts.
Implement DOM.getFrameOwner and DOM.describeNode to enable converting between DOM node handles and Frame handles. This would improve the implementation of Locator.contentFrame() and FrameLocator.owner(), as the current workaround using contentWindow does not work with shadow DOM, as seen in the page/frame-frame-element.spec.ts tests.
Introduce an event or method to track when a navigation is committed, beyond the current DOMContentLoaded and load events, which would be useful for some client requirements (e.g., 'should return when navigation is committed if commit is specified'). We need to know immediately if the navigation has failed or succeeded.
Interception is prohibitively expensive and therefore impractical.
Also not all requests are intercepted, e.g. WebSocket.
Add support for setting extra HTTP headers per user context, same as user agent
Allow to call script.addPreloadScript per context, so that the script is always evaluated before navigation in the page. See e.g. 'init script should run only once in popup'test in page-add-init-script.spec.ts.
Support offline mode emulation
Support emulation of disabled JavaScript
These are just some of the issue that we discovered while looking at the first cut. There are many other issues that are currently blocked by the ones above.
Implementation
Firefox: browsingContext.setViewport fails intermittently with {"type":"error","id":14,"error":"unknown error","message":"AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved","stacktrace":""}. See this bug report.
Firefox: page.setContent doesn't work because of 'Error: page.setContent: SecurityError: The operation is insecure.' in utility context. Bug report
Firefox pointerDown/pointerUp/pointerDown/pointerUp in separate input.performActions commands don't generate dblclick event, see 'should dblclick the div' in page-mouse.spec.ts
Firefox: script.realmDestroyed comes after browsingContext.contextDestroyed
Firefox: script.evaluate exception have no details/message
Firefox: browsingContext.contextDestroyed comes before input.performActions response, see 'should not throw UnhandledPromiseRejection when page closes'. I.e. some events can come after browsingContext was destroyed.
Firefox: navigation id stays the same for different navigations, see 'should return from goto if new navigation is started'
Intermittent Error: Protocol error (script.evaluate): unknown error
Message: AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved - apparently fails if a command is sent before about:blank navigation finishes.
Firefox: lazy loading iframes are not reported (can be disabled in settings I think)
Firefox: setViewport does not affect window.screen.width/height, matchMedia, see 'should emulate device width'
Firefox: CSP tests are failing
Firefox: "url" not supported yet in network.continueRequest
Firefox: no requestfailed event when it's cancelled by the page, see 'should not throw if request was cancelled by the page' in
Firefox: 'should amend utf8 post data' - doesn't work with non-latin post data
Firefox: 'redirected requests should report overridden headers' - does not allow to override headers on redirects
Firefox: browsingContext.create sometimes hangs in parallel tests
Firefox: browsingContext.downloadWillBegin is not fired
Firefox: "Blocked request with id 15 not found" when sending network.continueWithAuth, authentication not working
Firefox: about:blank page is required for firefox to not close even in headless! (--silent didn't work)
Firefox: crash reporter dialog, can probably be fixed with settings.
Firefox: need canonical settings for testing browser
Chromium: Method 'network.setCacheBehavior' is not implemented.
The text was updated successfully, but these errors were encountered:
To mention it here as well... Thanks a lot for all the details @yury-s! This is pretty helpful. As stated on the BiDi issue already I've created a Google document that contains all the above details, for collaborative editing. If you could find the time to at least roughly prioritize those features that would be great. You have a better idea what's most important, but I'll also dig into and see what could be the main issues with Firefox specifically to get more tests passing. Thanks!
@whimboo I've assigned priorities to some items in the document. Since these are the first major issues we encountered during testing, and many are blockers for further progress, most are marked as P1 or P0. For clarity, I've added definition for the priority values used. Those are priorities from Playwright's perspective. For Browser implementation issues, I highlighted those that result in massive failures (and should be prioritized higher). Beyond that I'd defer prioritization of individual feature bugs to the browser maintainers.
We've spent a couple weeks adding support for Bidi to Playwright and running Playwright tests with the new protocol. The good news is there has been a large progress in Bidi support in Firefox and Chromium, many Playwright tests are now passing. However, there is still a long list of issues that are blocking Playwright from adopting Bidi as the browser automation protocol. At this point the number of bugs we identified is too large to file individual reports for each of them, instead we'll use this issue to describe some of the major problems that we've discovered.
🎉 Current stats are the following:
Spec
browsingContext.SetViewport
command can only be sent to an existing page, and there is no way to pause the load of a new page (e.g., opened via ctrl+click or a popup); these pages always load with the default viewport first.proxy
andacceptInsecureCerts
per user context rather than persession
, so that same browser process can be reused for testing with different options.isMobile
emulation to enable fixed layout mode.tests/page/page-click.spec.ts
.DOM.getFrameOwner
andDOM.describeNode
to enable converting between DOM node handles and Frame handles. This would improve the implementation ofLocator.contentFrame()
andFrameLocator.owner()
, as the current workaround usingcontentWindow
does not work with shadow DOM, as seen in thepage/frame-frame-element.spec.ts
tests.DOMContentLoaded
andload
events, which would be useful for some client requirements (e.g., 'should return when navigation is committed if commit is specified'). We need to know immediately if the navigation has failed or succeeded.request.resourceType
support (see issue: Support ResourceType in network requests w3c/webdriver-bidi#704).suggestedFileName
tobrowsingContext.downloadWillBegin
.accept-language
headersnavigator.language
script.addPreloadScript
per context, so that the script is always evaluated before navigation in the page. See e.g.'init script should run only once in popup'
test inpage-add-init-script.spec.ts
.These are just some of the issue that we discovered while looking at the first cut. There are many other issues that are currently blocked by the ones above.
Implementation
browsingContext.setViewport
fails intermittently with{"type":"error","id":14,"error":"unknown error","message":"AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved","stacktrace":""}
. See this bug report.page.setContent
doesn't work because of 'Error: page.setContent: SecurityError: The operation is insecure.' in utility context. Bug reportpointerDown/pointerUp/pointerDown/pointerUp
in separateinput.performActions
commands don't generatedblclick
event, see 'should dblclick the div' in page-mouse.spec.tsscript.realmDestroyed
comes afterbrowsingContext.contextDestroyed
script.evaluate
exception have no details/messagebrowsingContext.contextDestroyed
comes before input.performActions response, see 'should not throw UnhandledPromiseRejection when page closes'. I.e. some events can come after browsingContext was destroyed.Message: AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved - apparently fails if a command is sent before about:blank navigation finishes.
setViewport
does not affectwindow.screen.width/height
,matchMedia
, see 'should emulate device width'browsingContext.create
sometimes hangs in parallel testsbrowsingContext.downloadWillBegin
is not firedabout:blank
page is required for firefox to not close even in headless! (--silent didn't work)The text was updated successfully, but these errors were encountered: