GGML is a tensor library for machine learning that allows large models to run efficiently on commodity hardware, with built-in optimization algorithms, 16-bit float support, and integer quantization support. The library is optimized for Apple Silicon and also supports x86 architectures via AVX / AVX2 intrinsics. There are no third-party dependencies, and zero runtime memory allocations. The library is open source and licensed under the MIT license, and encourages contributors to try out new ideas and build wild demos. Contributors can either contribute to the codebase or sponsor the project financially. The company ggml.ai is seeking to hire full-time developers to help advance the idea of on-device inference.
http://ggml.ai