Introduction
Have you heard of Tensorly? Tensorly is a collaborative open-source project which aims at providing a high-level Python API for tensor manipulations, decompositions and regression. In a nutshell, tensor decompositions are (mostly) unsupervised machine learning/source separation techniques that extract meaningful patterns from multiway arrays. A good introduction survey by Kolda and Bader can be found here. Tensors are actually much more complex mathematical items, see tensor network for other links and perspectives.
The strength of Tensorly is its flexible backend system that makes it possible to run the same code using multiple array librairies, such as numpy, pytorch or mxnet, while being targeted at novice users, providing a simple and clear user interface. This means that Tensorly can easily bridge the most recent developments in high performance computing (e.g. how to contract tensors very efficiently on GPU) and various efficient implementations of tensor decompositions such as CP/PARAFAC or Tucker.
Why improving Tensorly
Tensorly is already a great toolbox featuring many tensor manipulations (such as permuting entries in multiway arrays), operations (such as the Khatri-Rao product or the MTTKRP) and tensor decomposition models (such as CP/PARAFAC, Tucker, Matrix Product State, PARAFAC2). However, the library currently still misses some important functionalities:
- the algorithms implemented for training these models are not state-of-the-art.
- several important options regarding initialization are missing
- there is basically at most two different (and non state-of-the-art) algorithms for training a tensor decomposition in Tensorly, while the literature has dozens for each model (and it most likely holds that none of these algorithms is the best in all circumstances). Furthermore, adding many new algorithms within the current code architecture would be very messy.
- there is no plug-and-play system. Indeed, suppose you have a tensor problem which ressembles a standard one implemented in Tensorly. However you have a specific constraint that needs customization of the algorithms. This is not supported at the moment and you need to tinker directly with the Tensorly source code.
Project goals
In the light of the above criticisms, the Tensoptly project has several main objectives:
- Rework the Tensorly code structure. In particular, a class-based model similar to scikit-learn will be pursed. This way, a model such as CP/PARAFAC would be a class, with several methods correspond to various algorithms used to train it. This will have positively impact the collaboration means (easier to add new algorithms), and favor a didactic code that can help users understand tensor decompositions better.
- Adding several state-of-the-art algorithms (efficient implementation of second-order methods, extrapolated and randomized ALS, sketching techniques…) and models (constrained CP/PARAFAC and constrained Tucker, coupled factorizations).
- Allowing easy customization of the optimization algorithms so that new methods can be implemented within the Tensorly framework. This can help comparing newest methods with older ones, and build a good and fair benchmark.
- Providing examples on real data through didactic jupyter notebooks.
Project team
The Tensoptly project is leaded by Jeremy Cohen, and funded by Inria. However the main workforce will be a recruited engineer, over a 2-year period, see Job Offer. The Tensoptly project will be carried out in close coordination with:
- The core development team of Tensorly (tensorly.org), NVIDIA (in particular Jean Kossaifi and Anima Anandkumar’s team), and the development community which spans various institutions such as Imperial College London, the California Institute of Technology, etc.
- Members of the project-team Panama hosting the project (Jeremy Cohen, Nancy Bertin, Axel Marmoret, possibly interns).
- The users and contributors, which includes, amongst others, several Inria teams (SIROCCO, EMPENN, DANTE) the core development team of Tensorly See Links. External contributions are very welcome.