You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to run several models in parallel using python's builtin threading module. However, when trying to solve, the nl_writer (repn>plugins>nl_writer.py) does not appear to be able to handle writing files in parallel. In particular, it uses a shared resource AMPLRepn.ActiveVisitor, and a locking system which blocks any sort of parallel access from happening:
If two threads attempt to write separate problem files at the same time, the first thread will set ActiveVisitor, while the other will fail at the assert statement, since ActiveVisitor was set by the first (since it is shared among threads).
Note that this problem won't necessarily show up all of the time, since it relies on two threads trying to write .nl files at the same time (can be a rare occurrence) - it's more likely to happen with many threads and/or large problem files (?)
I am not sure if there is any easy way around this - the way this is implemented does appear to be an intentional design decision.
Steps to reproduce the issue
Below is an example python script which runs into the error pretty much all the time on my machine.
importpyomo.environaspyoimportthreadingdefthread_function():
model=pyo.ConcreteModel()
# create a lot of variables and bind them to constraints# (large model - will spend a lot time trying to write the problem file)# ie. x0 = 1, x1 = 1, x2 = 1, ...foriinrange(200):
model.add_component(f'x{i}', pyo.Var(domain=pyo.NonNegativeReals))
model.add_component(f'c{i}', pyo.Constraint(expr=model.component(f'x{i}') ==1))
# solve the modelsolver=pyo.SolverFactory('ipopt')
solver.solve(model)
if__name__=='__main__':
threads=list()
# create 500 threads to get a greater chance of hitting the errorforiinrange(500):
x=threading.Thread(target=thread_function)
threads.append(x)
print(f"Thread {i} Start")
x.start()
forindex, threadinenumerate(threads):
thread.join()
print(f"Thread {index} Done")
print("Main Done")
Error Message
line 16, in thread_function
solver.solve(model)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 597, in solve
self._presolve(*args, **kwds)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/solver/shellcmd.py", line 224, in _presolve
OptSolver._presolve(self, *args, **kwds)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 706, in _presolve
) = self._convert_problem(
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/solvers.py", line 757, in _convert_problem
return convert_problem(
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/opt/base/convert.py", line 99, in convert_problem
problem_files, symbol_map = converter.apply(*tmp, **tmpkw)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/solvers/plugins/converter/model.py", line 184, in apply
(problem_filename, symbol_map_id) = instance.write(
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/core/base/block.py", line 1991, in write
(filename, smap) = problem_writer(self, filename, solver_capability, io_options)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 348, in __call__
info = self.write(model, FILE, ROWFILE, COLFILE, config=config)
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 391, in write
with _NLWriter_impl(ostream, rowstream, colstream, config) as impl:
File "/home/alma/.local/lib/python3.10/site-packages/pyomo/repn/plugins/nl_writer.py", line 548, in __enter__
assert AMPLRepn.ActiveVisitor is None
AssertionError
Information on your system
Pyomo version: 6.7.0
Python version: 3.11.7
Operating system: Windows (running WSL)
How Pyomo was installed: pip from within conda environment
The text was updated successfully, but these errors were encountered:
Have you tried switching to concurrent.futures package? It does process based parallelization rather than thread based (it is definitely more limiting but may work for your use case)
This is a known issue with the NLv2 writer (at least it was reported to me offline last week). I think I have a solution that should remove the need for the singleton without significantly impacting performance. In the meantime, there are a couple workarounds:
As @svkawtikwar pointed out, use process-based parallelism (instead of thread-based)
Summary
I am trying to run several models in parallel using python's builtin
threading
module. However, when trying to solve, thenl_writer
(repn>plugins>nl_writer.py
) does not appear to be able to handle writing files in parallel. In particular, it uses a shared resourceAMPLRepn.ActiveVisitor
, and a locking system which blocks any sort of parallel access from happening:If two threads attempt to write separate problem files at the same time, the first thread will set
ActiveVisitor
, while the other will fail at theassert
statement, sinceActiveVisitor
was set by the first (since it is shared among threads).Note that this problem won't necessarily show up all of the time, since it relies on two threads trying to write
.nl
files at the same time (can be a rare occurrence) - it's more likely to happen with many threads and/or large problem files (?)I am not sure if there is any easy way around this - the way this is implemented does appear to be an intentional design decision.
Steps to reproduce the issue
Below is an example python script which runs into the error pretty much all the time on my machine.
Error Message
Information on your system
Pyomo version: 6.7.0
Python version: 3.11.7
Operating system: Windows (running WSL)
How Pyomo was installed: pip from within conda environment
The text was updated successfully, but these errors were encountered: