Welcome to the TDP Installation Guide. This page will walk you through setting up a TDP cluster in a straightforward manner. The deployment relies on the official tdp-getting-started repository which provides scripts and configurations to launch a TDP environment on top of a cluster of 7 VMs.

This cluster aims to be used for testing purposes only. Refer to the Documentation for production-ready clusters.

This guide provides instructions for Ubuntu systems. It has also been tested with NixOS and MacOS. Only x86_64 systems have been used for now. Alternative Linux distributions, as well as Windows, shall work.


Hardware and Software

Before you start, ensure your system meets the following requirements:


  • CPU: 8 cores
  • Storage: Minimum 3 GB (NVMe SSDs are better for performance)
  • Memory: 32 GB

If required, update the machine resources assigned to the VMs in the ./inventory/group_vars/all.yml file of your cloned repository.


  • Git a version control system. Install it by running these commands:

    # Update the package list and install git
    sudo apt update && sudo apt install git
  • Python3 (version 3.6 or newer), a versatile programming language. Most systems come with Python pre-installed. To check and install if needed:

    # Display the version of Python
    python3 -V

    If not, install Python3 using the following command:

    # Update the package list and install python3
    sudo apt update && sudo apt install python3
  • venv (Python virtual environment), for managing Python dependencies like Ansible and JMESPath. Install venv:

    # Update the package list and install venv
    sudo apt update && sudo apt install python3-venv
  • VirtualBox (version 6.1.26 or newer), a powerful virtualization software. Install it:

    # Update the package list and install Virtualbox
    sudo apt update && sudo apt install virtualbox
  • Vagrant (version 2.2.19 or newer), virtual machine management for development:

    # Install the Hashicorp trusted key
    wget -O- | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
    # Install the Hashicorp source
    echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
    # Update the package list and install Vagrant
    sudo apt update && sudo apt install vagrant
  • Ansible (version 2.9 or newer) to provision VMs on VirtualBox. Install it with these commands:

    # Include the official project’s PPA (personal package archive)
    sudo apt-add-repository ppa:ansible/ansible
    # Update the package list and install Vagrant
    sudo apt update && sudo apt install ansible
  • Zip and Unzip popular tools for managing compressed files. Install them:

    # Update the package list and install zip and unzip
    sudo apt update && sudo apt install zip unzip
  • jq (JSON Processor), required for some scripts during deployment. Install jq:

    # Update the package list and install jq
    sudo apt update && sudo apt install jq

Deployment Steps

Follow these steps to deploy your TDP cluster:

Step 1: Clone the tdp-getting-started Project

Run the following commands to clone the project:

git clone
cd tdp-getting-started

Step 2: Set Up the Environment

Run this command to set up essential components, including collections, jar releases, and Vagrant:

./scripts/ -e extras -e prerequisites -e vagrant

Step 3: Activate the Virtual Environment

Activate the Python virtual environment by running:

source ./venv/bin/activate && source .env

Step 4: Launch Virtual Machines

Use this command to start your virtual machines:

vagrant up

Step 5: Configure TDP Prerequisites

Run this playbook to configure services like Chrony, CA, LDAP, KDC, and PostgreSQL:

ansible-playbook ansible_collections/tosit/tdp_prerequisites/playbooks/all.yml

Step 6: Deployment

You can deploy your TDP cluster using either the TDP lib CLI or Ansible Playbooks.

You can refer to the official tutorial for other deployment methods. Software prerequisites may vary, as well as the Step 2, typically involving adapting the version tag and switching from -r stable, the default value, to -r latest.

To Deploy with TDP lib CLI, run:

tdp deploy

To Deploy with Ansible Playbooks, use:

ansible-playbook ansible_collections/tosit/tdp/playbooks/meta/all.yml

For extra services, run specific playbooks for Livy, ZooKeeper, Kafka, etc.

Step 7: Configuration

After deployment, configure HDFS user home directories:

ansible-playbook ansible_collections/tosit/tdp/playbooks/utils/hdfs_user_homes.yml

Configure Ranger policies:

ansible-playbook ansible_collections/tosit/tdp/playbooks/utils/ranger_policies.yml

Deploy Knox Gateway:

ansible-playbook ansible_collections/tosit/tdp/playbooks/meta/knox.yml

Next Steps

Connecting to the Cluster

Access nodes via ssh, for example:

vagrant ssh edge-01

Accessing the User Interface (UI)

To access the UI, configure your host for SPNEGO with your web browser. We recommend configuring your /etc/hosts file for easier access to the UIs. For detailed instructions, visit our host configuration page.

Learning to Use TDP

Now that you have a cluster up and running, explore our documentation to learn how to manage and administer your cluster effectively. You can also follow tutorials on specific components like HDFS, YARN, Spark, Hive and HBase.