4,916
questions
0
votes
0
answers
39
views
Optimizing function using multiprocessing.Pool()
I'm new in the multiprocessing topic. I am using a powerful machine with a lot of cores and I would like to use all the available cores.
This is the original function, which resolves the task really ...
- 87
1
vote
3
answers
38
views
Python Multiprocessing - losing command prompt window
I am running the following Python program to start a few processes to run simultaneously. Each process will run for several seconds. This is being started from a command prompt. But if I interrupt the ...
- 337
0
votes
1
answer
56
views
Python Multiprocessing - is it creating two threads per process
This program starts two processes. It appears that there may actually be two threads created for each process. One, which is the mainline code of the spawned python program and the other being the ...
- 337
0
votes
0
answers
14
views
How to read multiple bin. file with the apreader and the multiprocessing library?
from the GitHub is a multiprocessing to open one file described (https://github.com/leonbohmann/apreader)
import multiprocessing as mp
if __name__ == '__main__': # this line has to be included!
# ...
0
votes
1
answer
50
views
Is this the correct way of using sessions with multiprocessing in a process safe way in Python?
I am going to implement requests with sessions in my application, but I have read some threads concerning issues with thread and process safety, but those arent exactlty recent.
My application run 800+...
- 151
1
vote
0
answers
20
views
how to share hash() function output with multiprocessing.Array in python 3?
I need to share some 64-bit integers in a python 3 multiprocessing runtime.
I tried to use multiprocessing.Array, and it works on linux, but fails on Windows.
The long type for multiprocessing.Array ...
- 397
1
vote
2
answers
42
views
Are there any strategies around editing dictionaries nested in shared dictionaries?
I'm trying to edit a dictionary inside a shared dictionary in separate processes but they're immutable. Are there any techniques that I might be missing on how to get around this?
Here's an example
...
0
votes
1
answer
47
views
Single thread file loading and multiprocessing
I have a big data file (a few GB up to a few ten GB), which I want to read and process in multiple threads in python.
My current approach to read the file in parts (let's say 100MB) and pass that ...
- 33
0
votes
0
answers
18
views
TypeError: cannot pickle '_thread.lock' object when passing a class function
I am submitting executors from a ProcessPoolExecutor inside my class by passing a class method:
class BackwardEnrichmentService:
def __init__(
self,
...
) -> None:
.....
- 29
-1
votes
0
answers
28
views
Does the pandas DataFrame.to_hdf() function use multiprocessing? It seems to be throwing a strange error in my code
I'm editing a program a coworker made that is compiled into an exe with pyinstaller and then run from the command line.
The purpose of the program is to process some data and write it to a .h5 file ...
- 143
1
vote
0
answers
38
views
How to prevent multi process pool slow down overtime?
We are trying to run multiple simulation tasks using a multiprocess pool in order to reduce the overall runtime compared to running each task individually in a series. At the beginning of the run CPU ...
- 11
1
vote
0
answers
23
views
Multiprocessing for CPU-bound task (ML inference) in Jupyter Notebook
I'm working on Windows, on a Jupyter Notebook in VSCode.
I have a list of sentences and I want to apply a Hugging Face model to each one. Each one takes like 1 second, so I figured I could accelerate ...
- 583
1
vote
1
answer
49
views
Python multiprocessing.pool apply_async callback function fails to re-order output data
I am using the multiprocessing.pool apply_async function to process some video data via a computer vision algorithm, and the current strategy to speed up the process, since the algorithm has no memory,...
- 11
0
votes
0
answers
34
views
Process.Join() is delayed?
It appears Process.Join() waits for something to happen before continuing?
I have multiple scrapers running in different processes. Here is one of them:
def scraper_1(urls, start_time, ret_list, ...
- 19
1
vote
1
answer
40
views
Running a python child process concurrently doesn't seem to be concurrent
I've been tasked to update an existing python program that currently runs as a single process, to run a child process concurrently. I'm new to python concurrency techniques and wrote a simple script ...
- 58