test: reducer CUDA kernel tests #3162

ManasviGoyal · 2024-06-21T12:06:42Z

fixes the error for EmptyArray case in reducers
adds all test_0115_generic_reducer_operation.py tests for axis=-1 only in tests-cuda
adds cuda reducer tests for checking block boundary cases for array_size = 3000

ianna

@ManasviGoyal - there are some issues with this:

tests-cuda/test_3162_cuda_generic_reducer_operation.py ...FFFFFFFFFFFFFF [ 40%]
FFFFFFFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFE                             [100%]

==================================== ERRORS ====================================
_____ ERROR at teardown of test_0115_generic_reducer_operation_EmptyArray ______

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x774d45102340>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1719228956.2746673
stop       = 1719228956.2760165
when       = 'teardown'

../../../anaconda3/lib/python3.11/site-packages/_pytest/runner.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../anaconda3/lib/python3.11/site-packages/_pytest/runner.py:262: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
        ihook      = <_HookCaller 'pytest_runtest_teardown'>
        item       = <Function test_0115_generic_reducer_operation_EmptyArray>
        kwds       = {'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
../../../anaconda3/lib/python3.11/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
        argname    = 'nextitem'
        args       = ()
        firstresult = False
        kwargs     = {'item': <Function test_0115_generic_reducer_operation_EmptyArray>, 'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
        self       = <_HookCaller 'pytest_runtest_teardown'>
../../../anaconda3/lib/python3.11/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = False
        hook_name  = 'pytest_runtest_teardown'
        kwargs     = {'item': <Function test_0115_generic_reducer_operation_EmptyArray>, 'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
        methods    = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/home/ianna/anaconda3/lib/python3.11/site-packa...odule '_pytest.threadexception' from '/home/ianna/anaconda3/lib/python3.11/site-packages/_pytest/threadexception.py'>>]
        self       = <_pytest.config.PytestPluginManager object at 0x774d60e97190>
../../../anaconda3/lib/python3.11/site-packages/_pytest/unraisableexception.py:93: in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
E               
E               Traceback (most recent call last):
E                 File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
E                 File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
E               cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encounter
ed

cm         = <_pytest.unraisableexception.catch_unraisable_exception object at 0x774d3078af50>
err_msg    = 'Exception ignored in'
msg        = 'Exception ignored in: \'cupy.cuda.memory.Memory.__dealloc__\'\n\nTraceback (most recent call last):\n  File "cupy_
bac...\ncupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered\n'

../../../anaconda3/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
--------------------------- Captured stderr teardown ---------------------------
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
...

dev/generate-tests.py

Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>

ManasviGoyal · 2024-06-24T12:31:26Z

@ManasviGoyal - there are some issues with this:

tests-cuda/test_3162_cuda_generic_reducer_operation.py ...FFFFFFFFFFFFFF [ 40%]
FFFFFFFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFE                             [100%]

==================================== ERRORS ====================================
_____ ERROR at teardown of test_0115_generic_reducer_operation_EmptyArray ______

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x774d45102340>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1719228956.2746673
stop       = 1719228956.2760165
when       = 'teardown'

../../../anaconda3/lib/python3.11/site-packages/_pytest/runner.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../anaconda3/lib/python3.11/site-packages/_pytest/runner.py:262: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
        ihook      = <_HookCaller 'pytest_runtest_teardown'>
        item       = <Function test_0115_generic_reducer_operation_EmptyArray>
        kwds       = {'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
../../../anaconda3/lib/python3.11/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
        argname    = 'nextitem'
        args       = ()
        firstresult = False
        kwargs     = {'item': <Function test_0115_generic_reducer_operation_EmptyArray>, 'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
        self       = <_HookCaller 'pytest_runtest_teardown'>
../../../anaconda3/lib/python3.11/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = False
        hook_name  = 'pytest_runtest_teardown'
        kwargs     = {'item': <Function test_0115_generic_reducer_operation_EmptyArray>, 'nextitem': <Function test_0115_generic_reducer_operation_IndexedOptionArray_1>}
        methods    = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/home/ianna/anaconda3/lib/python3.11/site-packa...odule '_pytest.threadexception' from '/home/ianna/anaconda3/lib/python3.11/site-packages/_pytest/threadexception.py'>>]
        self       = <_pytest.config.PytestPluginManager object at 0x774d60e97190>
../../../anaconda3/lib/python3.11/site-packages/_pytest/unraisableexception.py:93: in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
E               
E               Traceback (most recent call last):
E                 File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
E                 File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
E               cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encounter
ed

cm         = <_pytest.unraisableexception.catch_unraisable_exception object at 0x774d3078af50>
err_msg    = 'Exception ignored in'
msg        = 'Exception ignored in: \'cupy.cuda.memory.Memory.__dealloc__\'\n\nTraceback (most recent call last):\n  File "cupy_
bac...\ncupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered\n'

../../../anaconda3/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
--------------------------- Captured stderr teardown ---------------------------
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 570, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
...

@ianna Hi, I have already added the fix for this so there shouldn't be any error. Can you pull all the changes in this PR and test again?

ianna

@ManasviGoyal - all tests pass, thanks!

ManasviGoyal · 2024-06-24T15:03:15Z

@ManasviGoyal - all tests pass, thanks!

@ianna Great! This PR and #3136 can be merged once the MacOS issue in the CI is fixed. Thanks!

jpivarski · 2024-06-25T15:32:26Z

This PR has a lot of conflicts with #3136, which has now been merged. The conflicts are in the kernels themselves, so it's something @ManasviGoyal will have to look at.

ManasviGoyal · 2024-06-25T16:45:36Z

This PR has a lot of conflicts with #3136, which has now been merged. The conflicts are in the kernels themselves, so it's something @ManasviGoyal will have to look at.

@jpivarski I have fixed the conflicts. If the tests pass, this should be ready for merge.

jpivarski

This includes corrections to the kernels as new tests, so it looks like the new tests found and fixed some problems.

I just ran them all on my GPU and everything still works. I think this is ready to merge, so I'll merge it now.

ManasviGoyal and others added 30 commits May 29, 2024 11:28

feat: add tree reduction implementation of argmin and argmax

860868d

feat: add awkward_ListOffsetArray_reduce_local_outoffsets_64 kernel

3cdbd7e

test: integration tests for cuda

c1a846b

test: some more integration tests for cuda

7be3f98

feat: add awkward_reduce_count_64 kernel

98fb7ed

fix: indexing and indentation

0ed94ef

feat: add awkward_reduce_countnonzero kernel

02c03bc

feat: add reduce sum, min and max kernels

34fc82b

feat: add reduce prod and sum_int_bool

4e00f07

feat: add sum_bool and prod_bool kernels

b28a605

fix: use cpt.assert_allclose

9e7abc7

test: reducer integration tests

458165c

fix: typr conversion

c75cb79

fix: use atomic to avoid race conditions

427670c

fix: remove unnessary variable

127e035

fix: minor fixes

8dee2ae

Merge branch 'main' into ManasviGoyal/add-reducer-kernels

b957bee

fix: all reducer for atomics

896770f

fix: missing template

f3d1cdc

fix: remove complex

ef47ead

fix: atomicMin() for float 32 and indentation

c881f1d

fix: pass correct dtype of identity

38d30b9

fix: remove combinations test

51b0e15

fix: manage resources and disable failing test

7e7fdc4

fix: uncomment fixed test for slicing

1148b95

fix: correctly interpret typetracer array for cuda backend

8e926ab

fix: tests-spec error for bool

38d314d

fix: check for the backend of head

15068b6

Merge branch 'main' into ManasviGoyal/add-reducer-kernels

d864481

test: reducer CUDAkernel tests

6cf0919

ManasviGoyal temporarily deployed to docs June 21, 2024 12:18 — with GitHub Actions Inactive

test: add more reducer tests

347c4c9

ManasviGoyal temporarily deployed to docs June 21, 2024 13:41 — with GitHub Actions Inactive

test: add more reducer tests 2

bc7e1c0

ManasviGoyal temporarily deployed to docs June 21, 2024 14:07 — with GitHub Actions Inactive

Merge branch 'main' into ManasviGoyal/reducer-tests

d34a7b6

ianna temporarily deployed to docs June 21, 2024 15:48 — with GitHub Actions Inactive

ManasviGoyal added 2 commits June 24, 2024 12:02

fix: error for EmptyArray

b59e428

test: generic_reducer_operation and block_boundary

1cf8e0d

ManasviGoyal force-pushed the ManasviGoyal/reducer-tests branch from f79a076 to 1cf8e0d Compare June 24, 2024 10:14

ManasviGoyal temporarily deployed to docs June 24, 2024 10:21 — with GitHub Actions Inactive

ianna requested changes Jun 24, 2024

View reviewed changes

dev/generate-tests.py Outdated Show resolved Hide resolved

Update dev/generate-tests.py

8df6bd4

Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>

ManasviGoyal temporarily deployed to docs June 24, 2024 12:37 — with GitHub Actions Inactive

ianna self-requested a review June 24, 2024 14:17

ianna approved these changes Jun 24, 2024

View reviewed changes

ManasviGoyal marked this pull request as ready for review June 24, 2024 14:48

ManasviGoyal mentioned this pull request Jun 24, 2024

feat: add reduce kernels #3136

Merged

13 tasks

fix: merge conflit

71ab593

ManasviGoyal deployed to docs June 25, 2024 16:53 — with GitHub Actions View deployment

jpivarski approved these changes Jun 25, 2024

View reviewed changes

jpivarski merged commit a1da072 into main Jun 25, 2024
39 checks passed

jpivarski deleted the ManasviGoyal/reducer-tests branch June 25, 2024 17:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: reducer CUDA kernel tests #3162

test: reducer CUDA kernel tests #3162

ManasviGoyal commented Jun 21, 2024 •

edited

Loading

ianna left a comment

ManasviGoyal commented Jun 24, 2024

ianna left a comment

ManasviGoyal commented Jun 24, 2024 •

edited

Loading

jpivarski commented Jun 25, 2024

ManasviGoyal commented Jun 25, 2024

jpivarski left a comment

test: reducer CUDA kernel tests #3162

test: reducer CUDA kernel tests #3162

Conversation

ManasviGoyal commented Jun 21, 2024 • edited Loading

ianna left a comment

Choose a reason for hiding this comment

ManasviGoyal commented Jun 24, 2024

ianna left a comment

Choose a reason for hiding this comment

ManasviGoyal commented Jun 24, 2024 • edited Loading

jpivarski commented Jun 25, 2024

ManasviGoyal commented Jun 25, 2024

jpivarski left a comment

Choose a reason for hiding this comment

ManasviGoyal commented Jun 21, 2024 •

edited

Loading

ManasviGoyal commented Jun 24, 2024 •

edited

Loading