andras/great-ai
andras/great-ai
Watch
1
0
Fork
You've already forked great-ai
 0
Code Issues Pull requests 2 Projects Releases 4 Packages 1 Wiki Activity Actions

Correct automatic chunk sizing descriptions

Browse source
This commit is contained in:
Andras Schmelczer 2026-10-27 21:59:20 +00:00
commit 3b717b5cbd
3 changed files with 3 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

2
great_ai/utilities/parallel_map/parallel_map.py
View file

@ -101,7 +101,7 @@ def parallel_map(
input_values: An iterable of items that `func` is applied to. input_values: An iterable of items that `func` is applied to.
chunk_size: Tune the number of items processed in each step. Larger numbers chunk_size: Tune the number of items processed in each step. Larger numbers
result in smaller communication overhead but less parallelism at the start result in smaller communication overhead but less parallelism at the start
and end. If `chunk_size` has a `__len__` property, the `chunk_size` is and end. If `input_values` has a `__len__` method, the `chunk_size` is
calculated automatically if not given. calculated automatically if not given.
ignore_exceptions: Ignore chunks if `next()` raises an exception on ignore_exceptions: Ignore chunks if `next()` raises an exception on
`input_values`. And return `None` if `func` raised an exception in a worker `input_values`. And return `None` if `func` raised an exception in a worker

2
great_ai/utilities/parallel_map/simple_parallel_map.py
View file

@ -32,7 +32,7 @@ def simple_parallel_map(
input_values: An iterable of items that `func` is applied to. input_values: An iterable of items that `func` is applied to.
chunk_size: Tune the number of items processed in each step. Larger numbers chunk_size: Tune the number of items processed in each step. Larger numbers
result in smaller communication overhead but less parallelism at the start result in smaller communication overhead but less parallelism at the start
and end. If `chunk_size` has a `__len__` property, the `chunk_size` is and end. If `input_values` has a `__len__` method, the `chunk_size` is
calculated automatically if not given. calculated automatically if not given.
concurrency: Number of new processes to start. Shouldn't be too much more than concurrency: Number of new processes to start. Shouldn't be too much more than
the number of physical cores. the number of physical cores.

2
great_ai/utilities/parallel_map/threaded_parallel_map.py
View file

@ -99,7 +99,7 @@ def threaded_parallel_map(
input_values: An iterable of items that `func` is applied to. input_values: An iterable of items that `func` is applied to.
chunk_size: Tune the number of items processed in each step. Larger numbers chunk_size: Tune the number of items processed in each step. Larger numbers
result in smaller communication overhead but less parallelism at the start result in smaller communication overhead but less parallelism at the start
and end. If `chunk_size` has a `__len__` property, the `chunk_size` is and end. If `input_values` has a `__len__` method, the `chunk_size` is
calculated automatically if not given. calculated automatically if not given.
ignore_exceptions: Ignore chunks if `next()` raises an exception on ignore_exceptions: Ignore chunks if `next()` raises an exception on
`input_values`. And return `None` if `func` raised an exception in a worker `input_values`. And return `None` if `func` raised an exception in a worker