Quick Configuration Guide for the ILM Store
Version for Apache Hadoop
Introduction
Apache Hadoop is an open-source software project that enables distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, including a very high degree of fault tolerance. Rather than relying on high-end hardware, the resiliency of these clusters comes from the software's ability to detect and handle failures at the application layer.
[Note] This is a quick set-up guide. For comprehensive documentation, refer to the Installation & Configuration Guide for the ILM Store (from SPS 13).
SAP ILM Store with Hadoop Architecture
The SAP ILM Store architecture with Hadoop involves the following components:
- Archiving Client: Initiates the archiving process.
- Storage and Retention Service (SRS): Manages the storage and retention policies for ILM-enabled archive files.
- ILM Store: The central repository for storing archive files.
- HDFS (Hadoop Distributed File System): The underlying distributed file system where data is stored.
The connection between the ILM Store and HDFS is established via HTTP.
Hadoop File System (HDFS) differs from a database by storing data as flat files in a directory structure. HDFS requires the file's name and directory path for access.
Table 1: Main URI / Path Components of the ILM Store and HDFS
| ILM Store | Hadoop File System |
|---|---|
| Collection | Directory |
| Resource | File |
Resources within the ILM Store are stored in specific collections, creating resource URIs that are also used in the Hadoop File System. Benefits include better traceability of files in HDFS and flexible selection of resources for moving or cleaning.
[Caution] Once the ILM Store is set up and operations have started, avoid making changes to the settings, as this can lead to the loss of stored information.
Prerequisites
To configure the ILM Store, ensure the following:
- You are using an SAP NetWeaver system with SAP_BASIS 740 Support Package 13 or SAP_BASIS 750 Support Package 02 (or later recommended).
- The business function ILM_STOR is activated.
- A productive Hadoop cluster is accessible from your system landscape.
[Note] This documentation does not cover Hadoop cluster installation. Ensure HttpFS or WebHDFS services are installed and enabled on your Hadoop cluster. These services provide HTTP REST APIs for HDFS operations, and the ILM Store's Hadoop Connector supports connections to both. Refer to the official Apache Hadoop documentation for details.
2.1 Important SAP Notes
Ensure relevant SAP Notes for component BC-ILM-STO are implemented. SAP Note 2563024 provides a list of notes for this component.
Authorizations
Access to the ILM Store
To access the ILM Store and save files:
- Create a technical user with appropriate authorizations.
- Assign a role to the user, including the following authorizations:
- Authorization Object: SILMSTOR, Field: ACTVT, Value: 16 (Execute)
- Authorization Object: S_DATASET, Field: FILENAME, Value: *
- Authorization Object: S_DEVELOP, Field: PROGRAM, Value: CL_ILM_STOR_DATASET=========== CP, RILM_STOR_PUT_WORKER
- Authorization Object: S_CTS_ADMI, Field: ACTVT, Value: 6 (Delete), 33 (Read), 34 (Write)
- Authorization Object: S_CTS_SADM, Field: CTS_ADMFCT, Value: TABL
- Authorization Object: S_CTS_ADMI, Field: CTS_ADMFCT, Value: TABL
- Authorization Object: S_CTS_SADM, Field: CTS_ADMFCT, Value: TABL
ILM Store Administration
For installation, configuration, and testing, a role with authorization object SILMSTORAD and ACTVT values 02 (Change), 07 (Activate, Generate), and 39 (Check) is required.
Origin Customizing
The origin identifies the data source and is crucial for Store configuration.
4.1 Administrative Customizing
Configure administrative settings via transaction ILM_STOR_ADM_CUST or SAP NetWeaver Customizing:
- Navigate to Application Server > Basis Services > Information Lifecycle Management > ILM Store > Define Settings for Administrative Customizing.
- Click 'Create'.
- Enter the Client and Logical File Name.
- Specify a name and description for the new Administrative Origin.
The configuration involves defining details for the ILM Store, including the client and logical file name, and specifying the administrative origin and its description. A table shows the default DB Connections for various ILM Store components (e.g., TILM_STOR, TILM_STOR_AUDIT).
Add an Operational Origin, providing its name and description. For the TILM_STOR_BLOB entry, set the DB Connection to 'DEFAULT'.
4.2 Operational Customizing
Maintain additional properties by starting transaction ILM_STOR_OPR_CUST or navigating through SAP NetWeaver Customizing > Application Server > Basis Services > Information Lifecycle Management > ILM Store > Define Settings for Operational Customizing. Enter the client from Administrative Origin and execute.
4.3 Configuring Storage Connections
Maintain entries in transaction SM30 for table TILM_STOR_CUS. Example entries map the 'adk' origin to HDFS connections:
| Origin | Namespace | Property | Value |
|---|---|---|---|
| adk | DB | DBCON.TILM_STOR_BLOB | HADOOP_CONNECTION_ID |
| adk | DB | CONSYS.HADOOP_CONNECTI ON_ID | HADOOP |
4.4 Routing Table Configuration
In transaction SM30, use table TILM_STOR_O_ROUT. Create an entry specifying the SAP System ID, Client, and Data Source (your operational origin).
4.5 Class Factory Customizing
In transaction SM30, use table TILMSTOR_CF. Create an entry specifying the Interface (e.g., IF_ILM_STOR_DB_OPERATOR), API Version (Default Version), Parameters (HADOOP), and Implement. Class (e.g., CL_ILM_STOR_DB_OPERATOR_HDP).
Publishing the ILM Store
5.1 Create the ICF Node for the ILM Store
Use transaction SICF to create a new service under the ILM node. Configure the Service Name, Description, and Handler (e.g., CL_ILM_STOR_WD_REQUEST_HANDLER). Ensure the service is active. The Logon Data tab requires a user with ILM Store access authorizations.
5.2 Create the RFC Destination
Two RFC connections are required:
5.2.1 Connection to HDFS
Use transaction SM59 to create an HTTP connection (type G) to the external server. Configure the target host and port for the Hadoop HTTP Service. Specify the path prefix, typically in the format /webhdfs/v1/<hdp_usr_home_path>, where <hdp_usr_home_path> is the user's home directory on Hadoop (e.g., /user/).
5.2.2 Connection to the ILM Store
Use transaction SM59 again to create another HTTP connection (type G). In the Technical Settings, configure the Target Host, Service Number (port), and Path Prefix (e.g., /sap/bc/ilm/zilm_stor, matching the ICF node path). Perform a connection test.
[Note] If SRS and ILM Store are on different systems, set up the RFC destination in the SRS system similarly. The ILM Store system's RFC destination is primarily for testing.
Maintenance of Hadoop Preferences in ILM Store
Maintain connection parameters, authentication, and access methods for successful operation and data transfer.
6.1 Hadoop Connection Customizing
Configure Hadoop connections using transaction SM30 for table TILM_STOR_HDP_C. Ensure the Connection ID aligns with the TILM_STOR_CUS configuration. An entry typically includes a Connection ID (e.g., HADOOP_CONNECTION_ID) and an Authentication Method (e.g., PSEUDO_AUTHENTICATION).
Key properties include:
- Max Data Size: Maximum data transferrable per HTTP request, dependent on Hadoop HTTP service setup.
- Authentication Method: User access method for Hadoop, chosen from options like Pseudo Authentication or No Authentication.
6.2 Access Customizing
For Pseudo Authentication, use transaction SM30 for table TILM_STOR_HDP_PU. Create entries to map origins (data sources) to Hadoop users for connection and data transfer. The specified Hadoop user must exist in the Hadoop system.
6.3 Hadoop Origin-Dependent Connection Customizing
Configure origin-dependent connections using transaction SM30 for table TILM_STOR_HDP_CO. Align the Connection ID with the database connection name from TILM_STOR_CUS. Specify the RFC Destination for HDFS.
Storage and Retention Service
The Storage and Retention Service (SRS) is essential for storing ILM-enabled archive files in the ILM Store. SRS must be activated in the application system.
7.1 Activation of SRS
SRS can be activated locally on the application system or on a separate remote system via an HTTP connection.
For detailed information, consult the SAP Help Portal under SAP Information Lifecycle Management > Making SAP ILM Available > Providing Stores for SAP ILM > Configuring the Service for the Control of ILM Stores > Configuring Storage and Retention Service for ILM Stores.
7.2 Creation of the ILM Store in the SRS Administration
In transaction ILMSTOREADM, create a new entry for available SRS stores. Provide an ILM Store name, a Description, and the previously created HTTP Connection (RFC destination).
The created store can be used in ILM rule maintenance via transaction IRMPOL. Refer to the SAP Help Portal for details on ILM Retention Management, Editing ILM Policies, and Editing Retention Rules.
Security
Key security tasks for the ILM Store include:
- Ensuring data accessibility only by authorized owners.
- Maintaining data confidentiality during HTTP communication.
- Ensuring data integrity.
8.1 Authentication
The Hadoop Connector supports two authentication methods: NO_AUTHENTICATION and PSEUDO_AUTHENTICATION. Other authentication protocols can be implemented by registering them in table TILM_STOR_HDP_A and creating a corresponding BAdI implementation. Standard HTTP Basic authentication is also supported, with user credentials stored in the RFC destination.
8.2 Apache KNOX
Apache Knox Gateway acts as a REST API Gateway for Hadoop clusters, providing a single access point for REST interactions. It supports HTTP Basic authentication, which can be configured within the RFC destination using the NO_AUTHENTICATION method.
8.3 HttpFS over HTTPS
HTTPS can be enabled at the RFC destination level. Verify SSL certificate setup using transaction SM59 and ensure correct security certificate configuration in transaction STRUST.
Testing the ILM Store
After setup, test the ILM Store functionality.
9.1 Origin for Test Purposes
Use a test origin (e.g., 'archeb') for testing configurations. Start transaction SM30, enter table TILM_STOR_CUS, and create new entries for the test origin by copying existing ones (e.g., 'adk', 'archive'). Test reports are mandatory for verifying ILM Store functionality.
9.2 Test Reports
Use report RILM_STOR_TEST_PF_SINGLE to test ILM Store functionality. In the report selection, provide the RFC destination created during storage connection configuration and execute.
For checking data constellations and operations as per specification BC-ILM 3.1, use report RILM_STOR_TEST_AT. Provide the RFC destination created for the ILM Store connection and execute.
To clear table entries, use report RILM_STOR_TEST_CLEAR. To test Hadoop communication, use report RILM_STOR_TEST_HADOOP, providing the DB Connection Name and Custom Connection Type.
References
- ILM Store on the SAP Help Portal: Link
- ILM Store Guide: Installation and Configuration Guide for the ILM Store
- Hadoop Configuration Guide: Hadoop Connector Configuration Guide
Important Disclaimers and Legal Information
This document contains important disclaimers and legal information regarding hyperlinks, external video hosting, beta features, example code, and bias-free language. SAP is not liable for content on external websites linked via the icon. SAP does not guarantee the availability of third-party hosted videos. Experimental features are not for productive use and are subject to change. Example code is for illustrative purposes only and not warranted for correctness. SAP promotes bias-free language in its documentation.
