feat: implement run_in_parallel#408
Open
tiagogcampos wants to merge 8 commits into
Open
Conversation
📝 WalkthroughWalkthroughThis PR introduces a new ChangesRun-In-Parallel Feature
Sequence DiagramsequenceDiagram
participant Client as Client Code
participant BrowserAPI as Browser.run_in_parallel()
participant Semaphore as Semaphore (if limit set)
participant Tasks as Coroutines
participant Results as Result Collector
Client->>BrowserAPI: run_in_parallel(*coroutines)
alt max_parallel_tasks is set and < count
BrowserAPI->>Semaphore: Create semaphore(limit)
loop For each coroutine
BrowserAPI->>Semaphore: acquire()
Semaphore-->>BrowserAPI: permit
BrowserAPI->>Tasks: Start coroutine
Tasks-->>BrowserAPI: Running
end
loop As tasks complete
Tasks-->>BrowserAPI: Result or Exception
BrowserAPI->>Semaphore: release()
BrowserAPI->>Results: Store in order
end
else No limit or limit >= count
BrowserAPI->>Tasks: Start all coroutines concurrently
Tasks-->>BrowserAPI: Results (may complete out of order)
BrowserAPI->>Results: Collect in submission order
end
Results-->>Client: list[T] (order preserved)
alt If any task raised
Results-->>Client: Exception propagated
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~55 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/en/features/browser-management/tabs.md`:
- Around line 699-703: The example calls to scrape_page use a single URL
argument but the function signature requires (tab, url); update the example to
pass the tab reference along with each URL (e.g., scrape_page(tab,
'https://example.com/a')) or refactor the snippet to iterate over a list of
(tab, url) pairs or map URLs onto an existing tab variable before calling
scrape_page; ensure you update every call in the block (scrape_page) so the
first parameter is the tab object expected by the scrape_page(tab, url)
implementation.
In `@docs/pt/features/browser-management/tabs.md`:
- Around line 699-703: O exemplo chama scrape_page(...) com apenas o URL, mas a
função definida espera a assinatura scrape_page(tab, url); corrija chamadores
nas linhas do exemplo para fornecer ambos os argumentos (o objeto tab usado no
bloco de criação/limite de paralelismo e o URL) ou, alternativamente, altere a
definição de scrape_page para aceitar somente (url) e atualizar suas referências
internas; referencie a função scrape_page e ajuste as invocações no bloco que
monta as chamadas paralelas para corresponder à assinatura usada.
In `@docs/zh/features/browser-management/tabs.md`:
- Around line 690-704: The example calls scrape_page(...) but doesn't define it;
add a minimal async scrape_page function (e.g., async def scrape_page(url): ...
) above the snippet and use it in the run_in_parallel call, or clearly mark
scrape_page as a placeholder; update the docs around ChromiumOptions, Chrome,
and browser.run_in_parallel to show the minimal async implementation that
returns a value (or an explicit TODO comment) so the example is runnable without
errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b4fdf476-af45-47d1-bee1-b0cb5d977f18
📒 Files selected for processing (21)
README.mdREADME_zh.mddocs/en/features/browser-management/contexts.mddocs/en/features/browser-management/tabs.mddocs/en/features/configuration/browser-options.mddocs/en/features/core-concepts.mddocs/pt/features/browser-management/contexts.mddocs/pt/features/browser-management/tabs.mddocs/pt/features/configuration/browser-options.mddocs/pt/features/core-concepts.mddocs/zh/features/browser-management/contexts.mddocs/zh/features/browser-management/tabs.mddocs/zh/features/configuration/browser-options.mddocs/zh/features/core-concepts.mdpydoll/browser/chromium/base.pypydoll/browser/interfaces.pypydoll/browser/options.pypydoll/elements/web_element.pytests/test_browser/test_browser_base.pytests/test_browser/test_browser_options.pytests/test_core_integration.py
Comment on lines
+699
to
+703
| scrape_page('https://example.com/a'), | ||
| scrape_page('https://example.com/b'), | ||
| scrape_page('https://example.com/c'), | ||
| scrape_page('https://example.com/d'), | ||
| ) |
There was a problem hiding this comment.
Fix scrape_page call signature in throttling example.
Line 699–Line 703 call scrape_page() with one argument, but earlier (Line 238) it requires (tab, url). This example won’t run as written.
Suggested doc fix
async with Chrome(options=options) as browser:
- await browser.start()
+ initial_tab = await browser.start()
+ tabs = [initial_tab] + [await browser.new_tab() for _ in range(3)]
results = await browser.run_in_parallel(
- scrape_page('https://example.com/a'),
- scrape_page('https://example.com/b'),
- scrape_page('https://example.com/c'),
- scrape_page('https://example.com/d'),
+ *[
+ scrape_page(tab, url)
+ for tab, url in zip(
+ tabs,
+ [
+ 'https://example.com/a',
+ 'https://example.com/b',
+ 'https://example.com/c',
+ 'https://example.com/d',
+ ],
+ )
+ ]
)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/en/features/browser-management/tabs.md` around lines 699 - 703, The
example calls to scrape_page use a single URL argument but the function
signature requires (tab, url); update the example to pass the tab reference
along with each URL (e.g., scrape_page(tab, 'https://example.com/a')) or
refactor the snippet to iterate over a list of (tab, url) pairs or map URLs onto
an existing tab variable before calling scrape_page; ensure you update every
call in the block (scrape_page) so the first parameter is the tab object
expected by the scrape_page(tab, url) implementation.
Comment on lines
+699
to
+703
| scrape_page('https://example.com/a'), | ||
| scrape_page('https://example.com/b'), | ||
| scrape_page('https://example.com/c'), | ||
| scrape_page('https://example.com/d'), | ||
| ) |
There was a problem hiding this comment.
Corrija a assinatura de scrape_page no exemplo com limite de paralelismo.
Nas Line 699–Line 703, scrape_page() é chamado com 1 argumento, mas a função definida antes recebe (tab, url). O snippet falha ao executar.
Sugestão de ajuste
async with Chrome(options=options) as browser:
- await browser.start()
+ initial_tab = await browser.start()
+ tabs = [initial_tab] + [await browser.new_tab() for _ in range(3)]
results = await browser.run_in_parallel(
- scrape_page('https://example.com/a'),
- scrape_page('https://example.com/b'),
- scrape_page('https://example.com/c'),
- scrape_page('https://example.com/d'),
+ *[
+ scrape_page(tab, url)
+ for tab, url in zip(
+ tabs,
+ [
+ 'https://example.com/a',
+ 'https://example.com/b',
+ 'https://example.com/c',
+ 'https://example.com/d',
+ ],
+ )
+ ]
)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/pt/features/browser-management/tabs.md` around lines 699 - 703, O
exemplo chama scrape_page(...) com apenas o URL, mas a função definida espera a
assinatura scrape_page(tab, url); corrija chamadores nas linhas do exemplo para
fornecer ambos os argumentos (o objeto tab usado no bloco de criação/limite de
paralelismo e o URL) ou, alternativamente, altere a definição de scrape_page
para aceitar somente (url) e atualizar suas referências internas; referencie a
função scrape_page e ajuste as invocações no bloco que monta as chamadas
paralelas para corresponder à assinatura usada.
Comment on lines
+690
to
+704
| ```python | ||
| from pydoll.browser.options import ChromiumOptions | ||
|
|
||
| options = ChromiumOptions() | ||
| options.max_parallel_tasks = 3 # 每次最多运行 3 个协程 | ||
|
|
||
| async with Chrome(options=options) as browser: | ||
| await browser.start() | ||
| results = await browser.run_in_parallel( | ||
| scrape_page('https://example.com/a'), | ||
| scrape_page('https://example.com/b'), | ||
| scrape_page('https://example.com/c'), | ||
| scrape_page('https://example.com/d'), | ||
| ) | ||
| ``` |
There was a problem hiding this comment.
示例缺少可运行的 scrape_page 定义。
Line 699-703 调用了 scrape_page(url)，但该代码块内没有对应定义，读者直接复制会报错。建议补一个最小实现（或显式注明占位函数）。
Suggested doc patch
from pydoll.browser.options import ChromiumOptions
+async def scrape_page(url: str):
+ # 示例占位：按需替换为你的抓取逻辑
+ await asyncio.sleep(1)
+ return url
+
options = ChromiumOptions()
options.max_parallel_tasks = 3 # 每次最多运行 3 个协程🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/zh/features/browser-management/tabs.md` around lines 690 - 704, The
example calls scrape_page(...) but doesn't define it; add a minimal async
scrape_page function (e.g., async def scrape_page(url): ... ) above the snippet
and use it in the run_in_parallel call, or clearly mark scrape_page as a
placeholder; update the docs around ChromiumOptions, Chrome, and
browser.run_in_parallel to show the minimal async implementation that returns a
value (or an explicit TODO comment) so the example is runnable without errors.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Related Issue(s)
Closes #224
Type of Change
How Has This Been Tested?
Executed full test suite of tests
Testing Checklist
Screenshots
Implementation Details
API Changes
Additional Info
Checklist before requesting a review
poetry run task lintand fixed any issuespoetry run task testand all tests pass