diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md index 593a4ef8809..427a7749586 100644 --- a/docs/get-started/Velox.md +++ b/docs/get-started/Velox.md @@ -129,11 +129,22 @@ In this way, only the gluten-jar is needed to add to `spark..ex the static version using a clean docker image without any extra libraries installed ( [build in docker](../developers/velox-backend-build-in-docker.md) ). On host with some libraries like jemalloc installed, the script may crash with odd message. You may need to uninstall those libraries to get a clean host. We **strongly recommend** user to build Gluten in this way to avoid dependency lacking issue. -With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to -pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and -set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the +With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to +pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and +set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the libraries list from the third-party jar. +### Dynamic OpenSSL with FIPS Support + +For environments requiring FIPS compliance or dynamic OpenSSL linking, Gluten supports building with dynamically linked OpenSSL. Set the `VCPKG_DYNAMIC_OPENSSL=ON` environment variable during build: + +```bash +export VCPKG_DYNAMIC_OPENSSL=ON +./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON +``` + +This enables OpenSSL with FIPS features and requires OpenSSL shared libraries to be available on all executor nodes at runtime. For detailed configuration and deployment instructions, see [Dynamic OpenSSL with FIPS Support](VeloxDynamicOpenSSL.md). + # Remote storage support ## HDFS support diff --git a/docs/get-started/VeloxDynamicOpenSSL.md b/docs/get-started/VeloxDynamicOpenSSL.md new file mode 100644 index 00000000000..ca2adaf4637 --- /dev/null +++ b/docs/get-started/VeloxDynamicOpenSSL.md @@ -0,0 +1,54 @@ +--- +layout: page +title: Dynamic OpenSSL with FIPS Support in vcpkg Build +nav_order: 8 +parent: Getting-Started +--- + +# Dynamic OpenSSL with FIPS Support in vcpkg Build + +## Overview + +By default, Gluten's vcpkg build statically links OpenSSL into the native libraries (`libvelox.so` and `libgluten.so`). However, in environments requiring FIPS (Federal Information Processing Standards) compliance or dynamic OpenSSL linking, Gluten now supports building with dynamically linked OpenSSL. + +This feature is particularly useful for: +- **FIPS compliance**: Organizations requiring FIPS 140-2/140-3 validated cryptographic modules +- **Security updates**: Easier OpenSSL security patching without rebuilding Gluten +- **System integration**: Using system-provided OpenSSL libraries + +## Prerequisites + +- vcpkg-based build system (requires `enable_vcpkg=ON`) +- OpenSSL development libraries installed on the system +- Supported platforms: x86_64 and aarch64 Linux + +## Building with Dynamic OpenSSL + +### Basic Build Command + +To enable dynamic OpenSSL linking with FIPS support, set the `VCPKG_DYNAMIC_OPENSSL` environment variable: + +```bash +export VCPKG_DYNAMIC_OPENSSL=ON +./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON +``` + +## How It Works + +When `VCPKG_DYNAMIC_OPENSSL=ON` is set: + +1. **vcpkg Configuration**: The `dynamic-openssl` feature is enabled in `vcpkg.json`, which includes OpenSSL with FIPS support +2. **Triplet Override**: Custom vcpkg triplets (`x64-linux-avx.cmake` and `arm64-linux-neon.cmake`) detect the environment variable and switch OpenSSL from static to dynamic linkage +3. **Build Process**: The `init.sh` script adds the `--x-feature=dynamic-openssl` flag to vcpkg install +4. **Library Packaging**: OpenSSL shared libraries are excluded from static linking but must be available at runtime + +> **_NOTE:_** +At runtime, `LD_LIBRARY_PATH` must include the OS-provided OpenSSL libraries, including `libssl.so`, `libcrypto.so`, and the FIPS-certified `fips.so`. These libraries must be available and loadable; otherwise, Gluten will fail to start. +At the time of the Gluten 1.7 release, Gluten is built and tested with OpenSSL `3.5.2`. Users should ensure that the OpenSSL libraries (`libssl.so` and `libcrypto.so`) available at runtime are compatible with those used during linking. To minimize the risk of ABI or API compatibility issues, we recommend using the same major OpenSSL version for both build-time and runtime environments. + +## References + +- [OpenSSL FIPS Module](https://www.openssl.org/docs/fips.html) +- [FIPS 140-2 Standard](https://csrc.nist.gov/publications/detail/fips/140/2/final) +- [vcpkg Documentation](https://vcpkg.io/) +- [OpenSSL Package in vcpkg](https://github.com/microsoft/vcpkg/blob/2025.09.17/ports/openssl/vcpkg.json) \ No newline at end of file diff --git a/docs/get-started/build-guide.md b/docs/get-started/build-guide.md index cdfad12f67e..148bd25cce6 100644 --- a/docs/get-started/build-guide.md +++ b/docs/get-started/build-guide.md @@ -30,6 +30,14 @@ Please set them via `--`, e.g. `--build_type=Release`. | build_arrow | Build arrow java/cpp and install the libs in local. Can turn it OFF after first build. | ON | | spark_version | Build for specified version of Spark(3.3, 3.4, 3.5, 4.0, 4.1, ALL). `ALL` means build for all versions. | ALL | +### Environment variables for build +These environment variables can be set before running build scripts to control build behavior. + +| Variable | Description | Default | +|------------------------|-----------------------------------------------------------------------------------------------|---------| +| VCPKG_DYNAMIC_OPENSSL | Enable dynamic OpenSSL linking with FIPS support (requires `enable_vcpkg=ON`). See [Dynamic OpenSSL](VeloxDynamicOpenSSL.md) for details. | OFF | + + ### Velox build parameters for build-velox.sh Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.