Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions docs/get-started/Velox.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,11 +129,22 @@ In this way, only the gluten-jar is needed to add to `spark.<driver|executor>.ex
the static version using a clean docker image without any extra libraries installed ( [build in docker](../developers/velox-backend-build-in-docker.md) ). On host with
some libraries like jemalloc installed, the script may crash with odd message. You may need to uninstall those libraries to get a clean host. We **strongly recommend** user to build Gluten in this way to avoid dependency lacking issue.

With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to
pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and
set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the
With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to
pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and
set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the
libraries list from the third-party jar.

### Dynamic OpenSSL with FIPS Support

For environments requiring FIPS compliance or dynamic OpenSSL linking, Gluten supports building with dynamically linked OpenSSL. Set the `VCPKG_DYNAMIC_OPENSSL=ON` environment variable during build:

```bash
export VCPKG_DYNAMIC_OPENSSL=ON
./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON
```

This enables OpenSSL with FIPS features and requires OpenSSL shared libraries to be available on all executor nodes at runtime. For detailed configuration and deployment instructions, see [Dynamic OpenSSL with FIPS Support](VeloxDynamicOpenSSL.md).

# Remote storage support

## HDFS support
Expand Down
49 changes: 49 additions & 0 deletions docs/get-started/VeloxDynamicOpenSSL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
layout: page
title: Dynamic OpenSSL with FIPS Support in vcpkg Build
nav_order: 8
parent: Getting-Started
---

# Dynamic OpenSSL with FIPS Support in vcpkg Build

## Overview

By default, Gluten's vcpkg build statically links OpenSSL into the native libraries (`libvelox.so` and `libgluten.so`). However, in environments requiring FIPS (Federal Information Processing Standards) compliance or dynamic OpenSSL linking, Gluten now supports building with dynamically linked OpenSSL.

This feature is particularly useful for:
- **FIPS compliance**: Organizations requiring FIPS 140-2/140-3 validated cryptographic modules
- **Security updates**: Easier OpenSSL security patching without rebuilding Gluten
- **System integration**: Using system-provided OpenSSL libraries

## Prerequisites

- vcpkg-based build system (requires `enable_vcpkg=ON`)
- OpenSSL development libraries installed on the system
- Supported platforms: x86_64 and aarch64 Linux

## Building with Dynamic OpenSSL

### Basic Build Command

To enable dynamic OpenSSL linking with FIPS support, set the `VCPKG_DYNAMIC_OPENSSL` environment variable:

```bash
export VCPKG_DYNAMIC_OPENSSL=ON
./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON
```

## How It Works

When `VCPKG_DYNAMIC_OPENSSL=ON` is set:

1. **vcpkg Configuration**: The `dynamic-openssl` feature is enabled in `vcpkg.json`, which includes OpenSSL with FIPS support
2. **Triplet Override**: Custom vcpkg triplets (`x64-linux-avx.cmake` and `arm64-linux-neon.cmake`) detect the environment variable and switch OpenSSL from static to dynamic linkage
3. **Build Process**: The `init.sh` script adds the `--x-feature=dynamic-openssl` flag to vcpkg install
4. **Library Packaging**: OpenSSL shared libraries are excluded from static linking but must be available at runtime
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add the following explicit clarification for users?

  • At runtime, LD_LIBRARY_PATH should point to the OS-provided OpenSSL package, which includes libssl.so, libcrypto.so, and the FIPS-certified fips.so, so that these libraries are loaded instead of the ones built at compile time by vcpkg.

  • Users should ensure compatibility between the OpenSSL libraries (libssl.so and libcrypto.so) used at link time and those available at runtime. We recommend using the same major version to avoid potential compatibility issues.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhouyuan can you also add the openssl version in vcpkg here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@philo-he @FelixYBW Thanks, I add more notes to callout the version and usage


## References

- [OpenSSL FIPS Module](https://www.openssl.org/docs/fips.html)
- [FIPS 140-2 Standard](https://csrc.nist.gov/publications/detail/fips/140/2/final)
- [vcpkg Documentation](https://vcpkg.io/)
8 changes: 8 additions & 0 deletions docs/get-started/build-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ Please set them via `--`, e.g. `--build_type=Release`.
| build_arrow | Build arrow java/cpp and install the libs in local. Can turn it OFF after first build. | ON |
| spark_version | Build for specified version of Spark(3.3, 3.4, 3.5, 4.0, 4.1, ALL). `ALL` means build for all versions. | ALL |

### Environment variables for build
These environment variables can be set before running build scripts to control build behavior.

| Variable | Description | Default |
|------------------------|-----------------------------------------------------------------------------------------------|---------|
| VCPKG_DYNAMIC_OPENSSL | Enable dynamic OpenSSL linking with FIPS support (requires `enable_vcpkg=ON`). See [Dynamic OpenSSL](VeloxDynamicOpenSSL.md) for details. | OFF |


### Velox build parameters for build-velox.sh
Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.

Expand Down
Loading