ValueError: Too large work array required -- computation cannot be performed with standard 32-bit LAPACK.

Dear Scipy community,

I am trying to run an SVD on a large matrix and I get the following error:

ValueError: Too large work array required – computation cannot be performed with standard 32-bit LAPACK.

My code:

import numpy as np
from scipy.linalg import svd

a = np.ones((30000,30000))
u,s,vh = svd(a)

import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)

returns:

(‘1.1.0’, ‘1.15.4’, sys.version_info(major=2, minor=7, micro=15, releaselevel=‘final’, serial=0))

I would expect to be able to run an SVD on a 30000 by 30000 matrix.

import platform
platform.architecture()

returns:

(‘64bit’, ‘’)

UPDATE:

It seems that svd(a, lapack_driver='gesvd') works, however I do not understand why the standard svd(a) with the deafult driver gesdd fails.

Issue Analytics

State:
Created 4 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

1reaction

AidanGGcommented, Jun 23, 2019

Yes, gesvd will work because the QR algorithm it uses for the bidiagonal SVD stage does not require as much working space.

Since the QR algorithm is much slower than the bidiagonal divide and conquer (BDC) of gesdd, I suggested the alternatives for using BDC if you really wanted to use it. Otherwise, you can continue to use gesvd.

For shared memory systems:

LAPACK/LAPACKE through MKL with 64-bit ints, or OpenBLAS built with INTERFACE64=1. Other vendors may offer 64-bit interfaces as well (Cray etc.) (Fortran/C)
Eigen through the BDCSVD implementation (C++)

For distributed memory systems, using MPI:

ScaLAPACK, again with 64-bit ints (Fortran/C)
Elemental, built with 64-bit ints and backed by a BLAS/LAPACK implementation built with 64-bit ints, as above (C++)

0reactions

AidanGGcommented, Jun 28, 2019

I believe that according to this issue https://github.com/numpy/numpy/issues/5906 it will not, but someone more knowledgeable than me regarding how numpy and scipy work will have to confirm.

Top Results From Across the Web

ValueError: Too large work array required -- computation ...

The LAPACK C lib has 32 bit long signed integers assigned, but your error says you need to work with longer integers.

computation cannot be performed with standard 32-bit LAPACK

Too large work array required — computation cannot be performed with standard 32-bit LAPACK_屎山搬运工的博客-CSDN博客

Python Numba or NumPy: understand the differences

... like the following failure: “ValueError: Too large work array required — computation cannot be performed with standard 32-bit LAPACK.”.

lapack.py | searchcode

17 However this requires the array to satisfy two conditions 18 which are memory ... computation cannot " 975 "be performed with standard...

scipy.linalg.lapack

... lwork > _np.iinfo(_np.int32).max)): raise ValueError("Too large work array required -- computation cannot " "be performed with standard 32-bit LAPACK.