vizibly diffrnt

Automating Content Management for Tableau Server

2017-12-26T00:00:00-08:00

Earlier this month I spoke at the Tableau Fringe Festival about Automating Content Management for Tableau Server. If you missed the webinar or want to rewatch it, here’s the video from my session.

You can watch all of the other amazing sessions on the Tableau Fringe Festival’s official YouTube channel.

Taking Tableau Further: Create Your own Content Management Tool, Part 3

2017-08-28T00:00:00-07:00

The Last Piece of the Puzzle

In the final part of this series, we’ll build upon the data we pulled from Tableau’s Postgres DB in part two to complete our Tableau Server Content Management Tool. This post will focus on how to use Tableau Server’s APIs to automate the process and regain valuable system resources. To complete our tool, we’ll be using the REST API and the lesser known Vizportal API. Before we being, let’s review our feature set…

1) Check for all workbooks and datasources that have not been used in N days.

2) Be able to “Whitelist” content that should not be altered. We can do this by “tagging” content directly on server and then reading those tags via the REST API.

3) Based on periods of inactivity and current schedules, decide what course of action to take: * Disable the Extract Schedule * Remove the Workbook/Datasource from Server * Reschedule the Extract to a More Efficient Time Slot

4) Send the Content Owner an email explaining the removal of their content from Server and include the .twb or .tds file as an attachment. If a thumbnail image of the Workbook is available, include that as well.

Setting up the Config

In the last post I introduced a single yaml configuration file that controls the settings needed for our tool to run. To enable our remaining features, we need to add a few more settings to it:

# Location for downloading Workbooks and Datasources from Tableau
download_settings:
    download_path: '/Path/To/Where/Tableau/Content/Will/Be/Downloaded/'

# Settings for our email delivery of content messages
email_settings:
    test_email: 'test@server.com' # A valid email address you can use for testing
    smtp_server: 'smtp.host.com' # The SMTP hostname for your email provider
    smtp_user: 'name@server.com' # A valid email that you can send messages from
    smtp_pw: 'password'
    notify_users: False # We'll use this as an override flag for easily disabling emails

# These settings control how we rebalance extract schedules. I'll go into more detail on this later but with this setup we can easily change our thresholds for content management
schedule_settings:
    180:
        Any: 'Disable'
    90:
        Any: 'Monthly'
    30:
        Weekly: 'Monthly'
        Daily: 'Weekly'
        Hourly: 'Weekly'
    14:
        Any: 'Weekly'
    7:
        Hourly: 'Daily'
    2:
        Any: 'Daily'

Now comes the fun part. Since most of what we’re looking to do is possible with the REST API we’ll start there.

Tableau Server REST API

Tableau Server ships with a really powerful set of API endpoints that can manage almost everything on our feature list.

Since we’re going to be using our config settings through out, make sure you include this block at the top of your code. I also recommend that you use an account with Server Admin privileges so you don’t run into any permissions issues.

import os, sys
from operator import itemgetter
import requests
import xml.etree.ElementTree as ET
import zipfile
import shutil
import logging #If you choose to enable logging, this library makes it really simple
import yaml

# import settings from yaml config
with open("config.yml", 'r') as ymlfile:
    cfg = yaml.load(ymlfile)

tab_server_settings = cfg["tab_server_settings"]
download_settings = cfg["download_settings"]
email_settings = cfg["email_settings"]
schedule_settings = cfg["schedule_settings"]

tab_server_url = tab_server_settings['tab_server_url']
tableau_username = tab_server_settings['tab_server_url']
tableau_user_pw = tab_server_settings['tableau_user_pw']

Sign In/Sign Out of Tableau Server

Before we can manage anything, we need to authenticate and sign into Tableau Server.

Detecting Your Tableau Server & API Versions

The REST API requires that we know the version of the API we’re using and that differs depending on what version of Tableau Server you are running. You could detect this programmatically or you can refer to the chart found here. To simplify things, we’ll hardcode this for now:

api_version = '2.6' # The latest version as of Tableau Server 10.3

# We'll need this to properly parse the XML responses from our API calls
if api_version <= '2.3':
    xmlns = {'t': 'http://tableausoftware.com/api'}
else:
    xmlns = {'t': 'http://tableau.com/api'}

We’ll also need to be able to parse our XML responses without running into issues with unicode characters. If you’ve seen the REST API examples from Tableau, this will look familiar:

def _encode_for_display(text):
    """
    Encodes strings so they can display as ASCII in a Windows terminal window.
    This function also encodes strings for processing by xml.etree.ElementTree functions.
    Returns an ASCII-encoded version of the text.
    Unicode characters are converted to ASCII placeholders (for example, "?").
    """
    return text.encode('ascii', errors="backslashreplace").decode('utf-8')

Now we can define our sign_in and sign_out functions:

This will return the token, site_id, and user_id that we’ll need to make our other REST API calls.

def sign_in(name, password, site=""):
    global token
    url = tab_server_url + "api/%s/auth/signin" % (api_version)

    #Build the request
    xml_payload_for_request = ET.Element('tsRequest')
    credentials_element = ET.SubElement(xml_payload_for_request,'credentials',name=name, password=password)
    site_element = ET.SubElement(credentials_element, 'site', contentUrl=site)
    xml_payload_for_request = ET.tostring(xml_payload_for_request)

    #Sign in to Tableau Server
    server_response = requests.post(url, data=xml_payload_for_request)
    if server_response.status_code != 200:
        print "Problem signing into Tableau Server: %s" % (server_response.text)
    else:
        print "Sign in to Tableau Server: %s" % (url)
    
    xml_response = ET.fromstring(server_response.text)

    #Retrieve the token and Site ID (we'll need these for the subsequent API calls)
    token = xml_response.find('t:credentials', namespaces=xmlns).attrib.get('token')
    site_id = xml_response.find('.//t:site', namespaces=xmlns).attrib.get('id')
    user_id = xml_response.find('.//t:user', namespaces=xmlns).attrib.get('id')

    print "Sign in to Tableau Server successful!\n"
    print "token = "+token, "site_id = "+site_id, "user_id = "+user_id+"\n"
    return token, site_id, user_id

Sign Out

When we’re done working with Tableau Server we need to log out and end our session.

def sign_out():
    global token
    url = tab_server_url + "/api/%s/auth/signout" % (api_version)
    headers = {'x-tableau-auth':token}
    server_response = requests.post(url, headers=headers)
    token = None
    if server_response.status_code != 200:
        print server_response.text
    else:
        print "Signed out of Tableau Server successfully.\n"
    return

A Note about Switching Sites

One of the most powerful features of Tableau Server is the ability to stand up separate Sites. Unfortunately, this can make managing content a bit of a bear. Workbooks and Datasources are not common across Sites and if you need to manage the content for your entire Server, you’ll probably need to do so on a Site-by-Site basis. When you sign in to Tableau Server, note how the calls require an element called the contentUrl. This is related to the Site you are logging into. In the example above, we left this blank which signs us into the Default Site. Beginning with version 10.3, Tableau has included a Switch Site endpoint which will allow you to change Sites without providing your user name and password again. If you’re running a version older than 10.3, you’ll need to repeat the Sign In step for every Site you want to manage content for.

Query Content on Server

Using the token and site_id from the sign_in function, we can now request content details from the server. We’ll keep parts of this call generic so we can use it as a template for building our other functions.

Note: This function contains 2 different endpoints, one for Workbooks and one for Datasources. Make sure you specify the correct content_type when using it.

def query_content(content_type, site_id, content_luid):
    """
    Queries the server for a specific piece of content.

    'content_type'  is the type of content to be queried. Acceptable values are 'workbook' and 'datasource'.
    'site_id'       is the ID (as a string) of the site on the server where the content is located. Use the value retrieved from sign_in()
    'content_luid'  is the ID (as a string) of the content on the server to to be queried.
                    Note that this is not the same as the workbook_id or datasource_id found in the Postgres database.
                    It is the unique id or luid value.

    Returns the name of the content and its owner_id (as a string).
    """
    headers = {'x-tableau-auth':token}

    # Configure call for querying Workbook
    if content_type == 'workbook':
        workbook_url = tab_server_url + "/api/%s/sites/%s/workbooks/%s" % (api_version,site_id,content_luid)
        content_response = requests.get(workbook_url, headers=headers)
        if content_response.status_code != 200:
            print content_response.text

        xml_response = ET.fromstring(_encode_for_display(content_response.text))

        workbook_name = xml_response.find('.//t:workbook', namespaces=xmlns).attrib.get('name')
        workbook_owner_id = xml_response.find('.//t:owner', namespaces=xmlns).attrib.get('id')

        tags = xml_response.findall('.//t:tag', namespaces=xmlns)
        workbook_tags = []
        for t in tags:
            workbook_tags.append(t.attrib.get('label'))

        print "Workbook name = %s and is owned by %s" % (workbook_name, workbook_owner_id)
        return workbook_name, workbook_owner_id, workbook_tags

    # Configure call for querying Datasource
    elif content_type == 'datasource':
        datasource_url = tab_server_url + "/api/%s/sites/%s/datasources/%s" % (api_version,site_id,content_luid)
        content_response = requests.get(datasource_url, headers=headers)
        if content_response.status_code != 200:
            print content_response.text
        xml_response = ET.fromstring(_encode_for_display(content_response.text))

        datasource_name = xml_response.find('.//t:datasource', namespaces=xmlns).attrib.get('name')
        datasource_owner_id = xml_response.find('.//t:owner', namespaces=xmlns).attrib.get('id')

        tags = xml_response.findall('.//t:tag', namespaces=xmlns)
        datasource_tags = []
        for t in tags:
            datasource_tags.append(t.attrib.get('label'))

        print "Datasource name = %s and is owned by %s" % (datasource_name, datasource_owner_id)
        return datasource_name, datasource_owner_id, datasource_tags

Download Content from Server

Using the query_content function as a template, we can build another function to download the content which we can then archive or send as an attachment to its owner. Here are the endpoints we’ll need for downloading Workbooks and Datasources.

GET /api/api-version/sites/site-id/workbooks/workbook-id/content

GET /api/api-version/sites/site-id/datasources/datasource-id/content

Remove Content from Server

Use the template again to build a function for removing content, this time using the DELETE option. Here is the format of the API call from Tableau’s API Reference.

DELETE /api/api-version/sites/site-id/workbooks/workbook-id

DELETE /api/api-version/sites/site-id/datasources/datasource-id

Unzip Content (optional)

If you downloaded your Workbooks as .twbx files or Datasources as .tdsx files, you might want to consider only sending the .twb or .tds when you email them to their creators to save on space. The easy way to do this is to rename the file as a .zip file and then extract the Workbook or Datasource file.

You can skip this step if you downloaded your content without the data.

def unzip_content(content_type, file_name, content_name):

    # Rename the file to .zip
    base = os.path.splitext(file_name)[0]
    os.rename(file_name, base + ".zip")
    zip_file_name = file_path + content_name + ".zip"

    #Extract zip file
    zip_ref = zipfile.ZipFile(zip_file_name,'r')
    #Location where extracted files will go
    if content_type == 'workbook':
        zip_ref.extractall(file_path+"Extracted Workbooks")
        zip_ref.close()
        print "Extract of .twb file complete\n"

        #Delete everything that's not the .twb file
        for f in os.listdir(file_path+"Extracted Workbooks"):
            if os.path.isdir(file_path+"Extracted Workbooks/"+f):
                shutil.rmtree(file_path+"Extracted Workbooks/"+f)
            elif not f.endswith('.twb'):
                os.remove(file_path+"Extracted Workbooks/"+f)
        print "Workbook cleanup complete\n"
        new_file = base.replace(download_path,'')+".twb"

    elif content_type == 'datasource':
        zip_ref.extractall(file_path+"Extracted Datasources")
        zip_ref.close()
        print "Extract of .tds file complete\n"

        #Delete everything that's not the .tds file
        for f in os.listdir(file_path+"Extracted Datasources"):
            if os.path.isdir(file_path+"Extracted Datasources/"+f):
                shutil.rmtree(file_path+"Extracted Datasources/"+f)
            elif not f.endswith('.tds'):
                os.remove(file_path+"Extracted Datasources/"+f)
        print "Datasource cleanup complete\n"
        new_file = base.replace(download_path,'')+".tds"

    #Delete the zip file
    os.remove(zip_file_name)
    print "File "+zip_file_name+" removed\n"
    return new_file

Download Workbook Images

To make our emails look professional, we can download a thumbnail of the preview image for a Workbook and include that with the .twb file.

GET /api/api-version/sites/site-id/workbooks/workbook-id/views/view-id/previewImage

Query Schedules on Server

And in order to modify content schedules, we’ll need to know what schedules are available on the server…

def query_schedules():
    global server_schedules
    headers = {'x-tableau-auth':token}
    schedule_url  = tab_server_url + "/api/%s/schedules" % (api_version)
    response = requests.get(schedule_url, headers=headers)
    if response.status_code != 200:
        print response.text

    xml_response = ET.fromstring(_encode_for_display(response.text))
    server_schedules = []

    schedules = xml_response.findall('.//t:schedule', namespaces=xmlns)
    for s in schedules:
        schedule_luid = s.attrib.get('id')
        schedule_name = s.attrib.get('name')
        schedule_state = s.attrib.get('state')
        schedule_frequency = s.attrib.get('frequency')
        schedule_type = s.attrib.get('type')
        schedule_details = {'schedule_luid':schedule_luid, 'schedule_name':schedule_name, 'state':schedule_state, 'frequency':schedule_frequency, 'type':schedule_type}
        server_schedules.append(schedule_details)

    return server_schedules

Tableau Vizportal API

While most of what we want to achieve is possible via the REST API, there are a couple of tasks that currently aren’t possible with it but are crucial to our tool, namely working with Extracts. For those we’ll use the Vizportal API.

Note: When using Vizportal, the ID values are not the same luid values that we use with the REST API. With Vizportal, they are the numeric ID that is found in the relevant table of the Postgres DB. Since we’re coupling these calls with the data we retrieved from Postgres, you should have what you need but you may need to validate the IDs if your calls aren’t working.

Signing into Vizportal

Signing in to Tableau Server via Vizportal is a lengthy process. I won’t go into it here but you can read all about how to do that via one of my prior posts.

Update Extract Schedules

Using the Vizportal API we can change the schedule associated with an extract, like moving something from a Daily refresh to once a Week. Pay close attention to the format of the payload variable. The Vizportal API is very particular about how these are structured.

def updateExtractSchedule(task_id, schedule_id, xsrf_token):
    payload = "{\"method\":\"setExtractTasksSchedule\",\"params\":{\"ids\":[\"%s\"], \"scheduleId\":\"%s\"}}" % (task_id, schedule_id)
    endpoint = "setExtractTasksSchedule"
    url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
    headers = {
    'content-type': "application/json;charset=UTF-8",
    'accept': "application/json, text/plain, */*",
    'cache-control': "no-cache",
    'X-XSRF-TOKEN':xsrf_token
    
    response = session.post(url, data=payload, headers=headers)
    if response.status_code != 200:
        print "Failed to update extract schedule"
    else:
        print "Extract schedule successfully updated"
    return response

Delete Extract Tasks

We can also use Vizportal to delete an Extract refresh task. This will not remove the content from your server, just the related Extract refresh.

def deleteExtractTasks(task_id, xsrf_token):
    payload = "{\"method\":\"deleteExtractTasks\",\"params\":{\"ids\":[\"%s\"]}}" % (task_id)
    endpoint = "deleteExtractTasks"
    url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
    headers = {
    'content-type': "application/json;charset=UTF-8",
    'accept': "application/json, text/plain, */*",
    'cache-control': "no-cache",
    'X-XSRF-TOKEN':xsrf_token
    response = session.post(url, data=payload, headers=headers)
    if response.status_code != 200:
        print "Failed to delete extract schedule"
    else:
        print "Extract schedule successfully removed"
    return response

Emailing Users

Outside of cleaning up our server, letting our users know why we’re doing this is the next most important item. For that we’ll send them an email detailing which content is being removed, the reason why and an attachment containing the .twb or .tds file. You can also include a 2nd attachment using the thumbnail image of the Workbook (if available).

import os, sys
from os, import path
import smtplib, ssl
import mimetypes
import email
import email.mime.application
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email.mime.text import MIMEText
from email.utils import formatdate
from email import encoders

run_mode = tab_server_settings['run_mode']
smtp_server = email_settings['smtp_server']
smtp_user = email_settings['smtp_user']
smtp_pw = email_settings['smtp_pw']
test_email = email_settings['test_email']

# Create a text/plain message
def send_email(From, To, Subject, Reason, FilePath="", FileName="", DaysSince=""):
    if run_mode == 'live':
        msg = email.mime.Multipart.MIMEMultipart()
        msg['Subject'] = Subject
        msg['From'] = From
        msg['To'] = To

        file_path = FilePath
        file_name = FileName
        if Reason == 'Remove':
            email_body = "Dear %s,\n\n Your content %s has been removed from Tableau Server because it has not been used in %s days. If you feel this has been removed in error, please notify your Tableau Server Admin or republish your content (attached).\n\n" % (To, FileName, DaysSince)
        elif Reason == 'Disable':
            email_body = "Dear %s,\n\n Your content %s has been disabled from updating on Tableau Server because it has not been used in %s days. If you feel this has been done in error, please notify your Tableau Server Admin or reschedule your content.\n\n" % (To, FileName, DaysSince)

        # The main body is just another attachment
        body = email.mime.Text.MIMEText(email_body)
        msg.attach(body)

        if Reason == 'Remove':
            part = MIMEBase('application', "octet-stream")
            part.set_payload(open(file_path+file_name, "rb").read())
            encoders.encode_base64(part)
            part.add_header('Content-Disposition', 'attachment; filename='+file_name)
            msg.attach(part)

        # send via SMTP server
        s = smtplib.SMTP(smtp_server)
        s.starttls()
        s.login(smtp_user,smtp_pw)
        s.sendmail(msg['From'],msg['To'], msg.as_string())
        s.quit()
        logging.info("Email sent to %s." % (To))
    else:
        print "DEMO MODE:Send email to %s with attached file: %s " % (To, FileName)

Connecting all of the Dots…

Now that we’ve defined all of our functions, it’s time to connect the dots and automate our process. To avoid scanning through all the content on the server and to make the script as efficient as possible, I use a Class to determine what action should be taken for each piece of content that meets the criteria we defined in tab_server_settings and schedule_settings.

Determine the Appropriate Action

class TableauContent(object):
"""
    Attributes:

    Required for Init:
    ------------------
    content_name: Name of the workbook or datasource
    content_type: workbook or datasource
    content_id: the unique id associated with the workbook or datasource
    task_id: the task_id related to the Extract refresh of the workbook or datasource
    current_schedule: the current schedule type (Hourly, Daily, Weekly, Monthly) of the Extract refresh
    current_schedule_id: the id associated with the current schedule

    Optional:
    --------
    new_schedule: the schedule type to change the Extract refresh to (Hourly, Daily, Weekly, Monthly)
    new_schedule_id: the id associated with the new schedule
    days_since: the days since a workbook was last viewed/datasource was last accessed. Default value is Zero if not supplied during Init
    action_needed: the action needed based on the criteria specified in sheriff_config.schedule_settings

"""
def __init__(self, content_name, content_type, content_id, site_id, task_id, current_schedule, current_schedule_id, new_schedule=None, new_schedule_id=None, days_since=0, action_needed=None):
    """
    Returns a Content object whose name is *content_name*
    """
    self.content_name = content_name
    self.content_type = content_type
    self.content_id = content_id
    self.site_id = site_id
    self.task_id = task_id
    self.current_schedule = current_schedule
    self.current_schedule_id = current_schedule_id
    self.new_schedule = new_schedule #Default to None
    self.new_schedule_id = new_schedule_id #Default to None
    self.days_since = days_since #Default set to Zero
    self.action_needed = action_needed #Default to None

def action(self):
    """
    Returns an expected action based on the days since a Workbook as last viewed or a Datasource was last accessed.
    """
    # Scan the schedule_settings from the imported config to find the action needed
    for k in sorted(schedule_settings.keys(),reverse=True):
        if self.days_since >= k:
            # Get the correct setting based on days_since
            ss = schedule_settings.get(k)
            # If there is more than one setting, pull the entry that matches the current_schedule
            if len(ss.keys()) > 1:
                for key,value in ss.items():
                    if self.current_schedule == key:
                        cs = key
                        ns = value
            # Otherwise use the entry that aligns with the days_since
            else:
                cs = ss.keys()[0]
                ns = ss.values()[0]
                action = ss.values()[0]

            break
        # If days_since does not exceed any limits in the config, apply the current settings and Do Nothing
        else:
            ns = 'Do Nothing'
            cs = self.current_schedule

    # Set the Action Needed to be Performed
    if ns == 'Disable':
        self.action_needed = 'Disable'
        self.new_schedule = None
        self.new_schedule_id = None
    elif ns == 'Do Nothing':
        self.action_needed = 'Do Nothing'
        self.new_schedule = self.current_schedule
        self.new_schedule_id = self.current_schedule_id
    else:
        self.action_needed = 'Reschedule'
        self.new_schedule = ns

    print "The action_needed for %s has been set to %s" % (self.content_name, self.action_needed)
    return self.action_needed

What this does is interpret the values from our schedule_settings to come up with an appropriate action based on the guidelines below. You can customize this by changing your yaml settings but I’ve found this to be good enough for even the largest servers.

Time Since Last Viewed/Accessed?	Current Schedule	Recommended Action
6 Months+	Any	Disable the Schedule
3 Months+	Any	Move to Monthly
1 Month+	Weekly	Move to Monthly
1 Month+	Hourly/Daily	Move to Weekly
2 Weeks+	Any	Move to Weekly
1 Week+	Hourly	Move to Daily
2 Days+	Hourly	Move to Daily

Designing the Workflow

All that’s left is to create our workflow and connect our functions. Here’s one you can follow for Workbooks.

Let’s Recap

There’s a bit more work needed to get everything running, but you should now have all the pieces needed to build a robust Content Management Tool for Tableau Server. Let’s recap what we’ve built:

We can now identify Workbooks and Datasources that have not been viewed/accessed for a given number of days. In our example, 180 days (6 Months).
We have all of the data and functions needed to loop through the Sites on our server and take the necessary action for a Workbook or Datasource.
We know who all of the content owners are and the email addresses associated with their server user names and built a mechanism for emailing them their content.
We have the list of all schedules available on Tableau Server and a way to modify the schedules related to our extracts.

If you made it this far, thanks for following along. This was a really fun tool to build and I hope you learned as much as I did. I’m excited to hear about how you’ll use this knowledge to build your own tools or automate your own Tableau Server needs. If you have any questions you can reach me through any of the channels listed here on my blog.

Happy Coding!

Taking Tableau Further: Create Your own Content Management Tool, Part 2

2017-06-23T00:00:00-07:00

Collecting the Building Blocks

In the first part of this series I talked about how we can build our own Content Management solution using Tableau’s APIs to add some functionality that will add life to your server. In this entry I’ll go into detail about which data points we need to retrieve from Tableau’s Postgres DB so we can build out our automation. I’ll also present some methods for constructing the code but won’t provide a full solution as you may have goals that differ from mine. What I want to leave you with are the concepts that can be applied more broadly.

Setting up the Config

I chose to design my tool using Python so we’ll use that for the examples here but you can use any language so long as there is a library that can connect to Postgres.

To save time and not be redundant, I’m going to use a central configuration file to manage all of the settings for our tool. I like using yaml for this because it’s really clean and easy to modify and works well with Python.

tab_server_settings:
	days_since_viewed: 180 #Setting the default to 180 days (6 months)
	days_since_accessed: 180
	tab_server_url: 'http://YourTableauServerURL'
	run_mode: 'live' #Could also be demo
	tableau_username: AdminUserName
	tableau_user_pw: AdminPassword
	remove_disabled_content: False
	
postgres_settings:
	tab_server_pg_db: 'tab_server_postgres_hostname'
	pg_user: pg_username #usually this is 'readonly'
	pg_pw: password #you specify this
	pg_database: 'workgroup' #this is the default for Tableau Server 
	pg_port_num: 8060 #default port
	

Save the file with .yml extension and make sure you have PyYAML installed. You can add more configs as you go but for now let’s keep things simple.

Postgres

The following assumes that you have access to Tableau Server’s Postgres Database.

Not every data point that we need (id values of extracts and content usage statistics) is available from Tableau’s APIs but because every Tableau Server ships with its own Postgres DB, we can go right to the source to get our data. It’s best to connect to a copy of the server database and not the live one to avoid impacting your server’s performance. I’m using the psycopg2 library to connect to the Workgroup database.

Connect to Postgres

import time
import os, sys
from os import path
import more_itertools
from  more_itertools import unique_everseen
from operator import itemgetter
import psycopg2
import sys
import pprint
import requests
import xml.etree.ElementTree as ET
import urllib
import zipfile
import shutil
import yaml
sys.path.append("..")

# import settings from yaml config

with open("config.yml", 'r') as ymlfile:
	cfg = yaml.load(ymlfile)

tab_server_settings = cfg["tab_server_settings"]
postgres_settings = cfg["postgres_settings"]

# Set variables based on Config File
run_mode = tab_server_settings['run_mode']

days_since_viewed = tab_server_settings['days_since_viewed']
days_since_accessed = tab_server_settings['days_since_accessed']

tab_server_pg_db = postgres_settings['tab_server_pg_db']
pg_user = postgres_settings['pg_user']
pg_pw = postgres_settings['pg_pw']
pg_database = postgres_settings['pg_database']
pg_port_num = postgres_settings['pg_port_num']

def pg_connect():
    """
    Signs in to the Postgres database specified in the global tab_server_pg_db variable.
    Returns a cursor with an open connection for querying tables in the workgroup database.
    """
    # Define our connection string
    global conn_string
    conn_string = "host='%s' dbname='%s' user='%s' password='%s' port='%s'" % (tab_server_pg_db,pg_database,pg_user,pg_pw,pg_port_num)

    # print the connection string we will use to connect
    print "Connecting to database\n	->host:%s, dbname:%s, port:%s" % (tab_server_pg_db, pg_database, pg_port_num)

    # get a connection, if a connect cannot be made an exception will be raised here
    global conn
    try:
        conn = psycopg2.connect(conn_string)
    except Exception as err:
        print "Unexpected error while establishing connection:{}".format(err)
        sys.exit(1)

    # conn.cursor will return a cursor object, you can use this cursor to perform queries
    global cursor
    cursor = conn.cursor()
    print "Connected!\n"

    return cursor

Once you open the connection, the cursor can be used to issue other commands to the database like this:

cursor.execute("SELECT [fields] from [table]")
data = cursor.fetchall()

When you’re done getting data from Postgres, close the connection like this:

conn.close()

Get Sites

Because we want to review all of the content on our server, regardless of Site, we’ll need to get the details for all Sites. Using the cursor you opened, write a function to query the sites table for the following data:

id: the numerical id value of the Site
name: the friendly name of the Site
url_namespace: also known as the Site ID or contentUrl. It is the value that appears after “/site” in the URL on Tableau Server
luid: the unique identifier of the Site (it is used as the “site_id” for API calls)

def get_pg_sites():
	cursor.execute("select id, name, url_namespace, luid from sites")
	sites = cursor.fetchall()
	global sites_data
	sites_data = []
	
	for row in sites:
		values = [row[0], row[1], row[2] row[3]]
		record = {'site_id':values[0], 'site_name':values[1], 'url_namespace':values[2], 'site_luid':values[3]}
		sites_data.append(record)
	
	return sites_data

The goal of the function above is to return a list where each item is a complete database “record”, formatted as a dictionary, with the field names as keys and their associated values. Use this function as a template for the remaining data to be retrieved, substituing the query and adding fields to the output as necessary.

List Unused Content

Since the goal of this tool is clean up content, we need to identify which content (Workbooks and Datasources) have become stale.

select id, name, workbook_url, owner_id, project_id, site_id, site_luid, luid, date_part('day',now()-last_view_time) as days_since_last_view 
from 
(
	select w.id, w.name, w.workbook_url, w.owner_id, w.project_id, w.site_id, wkb.luid, sites.luid as site_luid, max(vs.last_view_time) as last_view_time
	from _workbooks w
	left join _users u on w.owner_id=u.id
	left join _views_stats vs on w.id=vs.views_workbook_id
	join workbooks wkb on w.id=wkb.id
	join sites on sites.id=w.site_id
	where date_part('day',now()-last_view_time) >= %s
	group by w.id, w.name, w.workbook_url, w.owner_id, w.project_id, w.site_id, wkb.luid, site.luid
) a

Remember where we set days_since_viewed in the yaml configuration? Swapping out %s in the query with this variable will allow you to change the yaml as needed and not have to modify this query. As with the Site details before, store the records as a list of key, value pairs.

The same process needs to be done for Datasources but the tables and fields are different.

Get User Details

Once we know the content we want to examine, we need to know a little bit about the users that published it. Using the prior examples, write another function to retrieve the following:

select 
	su.friendly_name as user_friendly_name,
	su.name as system_user_name,
	su.email,
	u2.licensing_role_name,
	u2.site_id,
	u.luid as user_luid
from system_users su
join users u on su.id=u.system_user_id
join _user u2 on u.id=u2.id
where u.id = %s

This time, %s represents a substituted variable for a user_id. In this case, we’ll be passing an owner_id that we got from our list of Workbooks and Datasources so we don’t scan through all the users on the server.

Get Server Schedule Details

We’ll also need all of the possible schedules that exist on the server. Here’s the query:

select id, luid, name,
case schedule_type 
	when 0 then 'Hourly'
	when 1 then 'Daily'
	when 2 then 'Weekly'
	when 3 then 'Monthly' end as schedule_type
from schedules

Get Content Schedule Details

And last but not least, the schedules for particular pieces of content (since they can often have more than one). This query is pretty large because we need parse the dayofweekmask to determine which days during the week a piece of content is scheduled for.

select workbook_id, workbook_name, task_id, schedule_id, schedule_type, priority, 
day_of_week_mask_array[1] is_sat,
day_of_week_mask_array[2] is_fri,
day_of_week_mask_array[3] is_thu,
day_of_week_mask_array[4] is_wed,
day_of_week_mask_array[5] is_tue,
day_of_week_mask_array[6] is_mon,
day_of_week_mask_array[7] is_sun
from 
(
	select *, regexp_split_to_array(day_of_week_mask_binary,'') as day_of_week_mask_array
	from
	(
		select workbook_id, workbook_name, task_id, schedule_id,
		schedule_name, priority, 
		case schedule_type 
			when 0 then 'Hourly'
			when 1 then 'Daily'
			when 2 then 'Weekly'
			when 3 then 'Monthly' end as schedule_type,
		day_of_week_mask,
		cast(day_of_week_mask::bit(7) as varchar) as day_of_week_mask_binary,
		day_of_month_mask
		from
		(
			select w.id as workbook_id, w.name as workbook_name,
			w.owner_id, w.project_id, w.site_id, t.id as task_id,
			t.schedule_id, t.type, t.luid, s.name as schedule_name,
			s.priority, s.schedule_type, s.day_of_week_mask, s.day_of_month_mask
			from tasks t
			inner join _workbooks w on t.obj_id=w.id
			left join _schedules s on t.schedule_id=s.id
			where w.id = %s
		) a
	) b
) c

Again, substitute %s for a variable representing the Workbook Id. You’ll need to write a similar query for Datasources, substituting the Datasource Id as needed.

What is All of this Data Used For?

At this point you’re probably asking yourself:

“Why did I just write all of these functions and really long, complex queries?”

“What on Earth am I supposed to do with all of this?”

The simple answer is that we now have the building blocks for the rest of the API calls that we want to make. Let’s do a recap of what we know:

We can now identify Workbooks and Datasources that have not been viewed/accessed for a given number of days.
We have all of the details needed to loop through the Sites on our server and issue the proper commands via the REST API.
We know who all of the content owners are and the email addresses associated with their server user names. This will be hepful when we need to send them notifications.
We have the list of all schedules available on Tableau Server.
We have the schedule details for any Workbook or Datasource on server, including their frequency of execution.

Using these data points we can start to build our workflow and begin managing content based on the settings we supply in our yaml configuration.

In my next post I’ll cover how we structure the REST API calls and automate the workflow.

Taking Tableau Further: Create Your own Content Management Tool, Part 1

2017-06-13T00:00:00-07:00

Consider the lifecycle above as it relates to your Tableau content. You work hard to Plan, Develop, and Test your content and then you make it available to your audience by publishing it to Tableau Server. After your users sign off on it, it goes into Production where (hopefully) they use it to extract insights that impact their business.

But what happens next? How do you know if it’s being used? What if the question your visualization was built to answer is no longer relevant and your workbooks are going unused? How do you manage your content at scale?

With recent releases of Tableau Server, Tableau has expanded their Enterprise capabilities to tackle many of these questions. Content analytics provide a window into who is using your content and how frequently. Data driven alerts notify users of changes in metric values or when extracts fail to update. But what happens when people stop using your visualizations? Wouldn’t it be great if Tableau could intelligently detect content that’s no longer being used and manage it for you? Today there is no out-of-the-box way to do this effectively and at scale.

Content Analytics on Tableau Server

Your organization may have a retention policy or guidelines for scheduling content but today all of those rules need to be implemented manually by your admin team. And if you run a lean team that doesn’t have the bandwidth to regularly clean up 1000’s of workbooks the task can be daunting. Don’t despair, there is hope.

Using Tableau’s APIs (official and unofficial) we can create our own Content Management Tool that will allow us to meet our requirements, however unique they might be.

Determining Your Feature Set

Before we start coding it’s really important to know what features you want your Content Management tool to handle. This helps you focus on an outcome and avoid chasing after things that might only have marginal value. Through my client work I’ve seen a variety of content management challenges so I chose to focus here on ones that are applicable to a wide range of Tableau environments. For my solution, I chose the following:

1) Check for all workbooks and datasources that have not been used in N days (configurable based on need). This will form the basis for the actions we’ll take further on.

2) Be able to “Whitelist” content that should not be altered. This could be because it has significant operational need or because you know the C-Suite looks at it or because it took a really long time to build and you want everyone to know that. Just kidding about that last one. Any reason to preserve content will do.

3) Based on periods of activity, or inactivity, the tool should decide what course of action to take. Potential actions include:

Disable the Extract Schedule
Remove the Workbook/Datasource from Server
Reschedule the Extract to a More Efficient Time Slot

4) If a particular piece of content is to be removed, send the Content Owner an email explaining why their content is being removed and include their content as an attachment so it’s not lost forever. The attachment should be a .twb or .tds file to minimize the file size and the email should include a thumbnail image of the workbook if one is available.

5) Log all actions and produce a summary of each action taken for review, aka Demo Mode.

Putting Ideas into Action

Now that we know what features to build out, we need to figure out how to get the information we need. For this feature set we’ll need:

Access to Tableau Server’s Postgres database - so we can review content usage patterns and get scheduling details.
The Tableau Server REST API - for getting our whitelisted content, downloading workbooks and datasources, deleting content from the server.
The Tableau Vizportal API - for disabling extracts, changing extract schedules
Access to an SMTP Server for sending emails to users. Alternatively, you could subtitute email notifications for Slack or whatever your company uses for collaboration.

I chose to code my solution with Python but you could use any language that supports connections to Postgres and HTTP requests for working with Tableau’s APIs. Another thing to consider is who your audience is going to be. Are you designing for an experienced IT admin that knows how to modify a config file or could your end user be semi-technical analytics manager that would prefer a sleek web-app? These questions can impact your design decisions so think about how you want users to interact with your tool before you begin coding.

In the next part of this series I’ll dive into what data points we need to retrieve to enable our features and the design methodology I used to build this content management tool.

How to Detect your Tableau Server’s REST API Version

2017-03-03T00:00:00-08:00

With each release Tableau continues to improve the functionality of their REST API. This week we saw the release of Tableau 10.2 and with that came some enhancements that make working with the API easier:

The ability to specify the fields you want to receive from a GET request
CORS and JSON support for Javascript and web development
The ability to download high-resolution images of views (previously only available through tabcmd)

Of course these are only available with the latest version of the REST API (2.5 as of the release of 10.2). So what if you’re running a different version of Tableau Server? How do you know which version of the API your server supports?

Detecting the version of REST API supported by your Tableau Server

Starting with Tableau 10.1 (API version 2.4), Tableau included the endpoint ServerInfo that contains the version of the API supported by your Tableau Server in its response. It looks like this:

GET /api/api-version/serverinfo

and the response looks like this:

<tsResponse>
    <serverInfo>
        <productVersion build="build-number">product-version</productVersion>
        <restApiVersion>api-version</restApiVersion>
    </serverInfo>
</tsResponse>

Pretty great, right?

If you look closely, you’ll see that the call requires you to put the api-version in the URL. But wait…isn’t that what we’re trying to figure out? How can you input the API version when you don’t know what it is? In fact, how can you tell what version of Tableau Server you’re sending calls to?

By using the Vizportal API, that’s how.

Detecting your Tableau Server and API version using the Vizportal API

In most cases you’ll know what version of Tableau Server you’re using and can use that to determine your API Version with the table below:

Tableau Server version	REST API version	Schema version
8.3, 9.0	2.0	2.0
9.0.1 and later versions of 9.0	2.0	2.0.1
9.1	2.0	2.0.1
9.2	2.1	2.1
9.3	2.2	2.2
10.0	2.3	2.3
10.1	2.4	2.4
10.2	2.5	2.5

But what if you’re designing something that might be open-sourced to the Tableau Community or at a client site and you don’t want to hard-code those values in? With a quick call to the Vizportal API and some JSON parsing you can determine what version of Tableau Server you’re connecting to and map that its API Version. The code below builds on a previous post where I showed how you can log into the Vizportal API. We’ll start where that post left off.

# The information we want (Tableau Server major and minor version) is contained in the response from this call
# Re-use the xsrf_token you captured from the Vizportal login

def getSessionInfo(xsrf_token):
    payload = "{\"method\":\"getSessionInfo\",\"params\":{}}"
    endpoint = "getSessionInfo"
    url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
    headers = {
    'content-type': "application/json;charset=UTF-8",
    'accept': "application/json, text/plain, */*",
    'cache-control': "no-cache",
    'X-XSRF-TOKEN':xsrf_token
    }
    SessionResponse = session.post(url, data=payload, headers=headers)
    return SessionResponse

# We need to parse the JSON response from the getSessionInfo call
# and extract the major and minor version numbers

def DetectTableauVersion():
    global xsrf_token
    TabServerSession = getSessionInfo(xsrf_token).text
    TSS = json.loads(TabServerSession)
    v = TSS['result']['server']['version']['externalVersion']
    major = v['major']
    minor = v['minor']
    patch = v['patch']
    tsVersion = major+'.'+minor
    api_version = None

# Hit this endpoint to access a JSON version of the lookup table above
# Note: this endpoint is not provided by Tableau and might not be hosted in the future

    api_versions = requests.get('https://tbevdbgwch.execute-api.us-west-2.amazonaws.com/Production/versions/')
    api_lookup = json.loads(api_versions.text)
    for k,v in api_lookup['server'].iteritems():
        if k == tsVersion:
            api_version = v
    return tsVersion, api_version
    
tsVersion, api_version = DetectTableauVersion()

print "Tableau Server Version: %s" % (tsVersion)
print "API Version: %s" % (api_version)

Success! We now know what version of Tableau Server we’re using and what version of the API to include in any REST API calls. It’s a little clunky but it works if you’re unsure of your server’s version or if you’re using a version prior to 10.1.

The Unofficial Guide to Tableau’s Vizportal API

2017-01-26T00:00:00-08:00

Late last year I began experimenting with Tableau’s Undocumented API and I thought:

“Wow this was hard to figure out!

What if there was reference out there with all of these methods in it so other people didn’t have to dig through their browser’s code or run network traces and inspect the logs to see how these calls work?”

Good news! Now there is…

Presenting the Unofficial Guide to Tableau’s Vizportal API. My goal with this guide was to capture the majority of the calls you can perform using the Vizportal API and make it accessible to anyone. While Tableau offers a pretty robust REST API, there are a few things that haven’t made it in there that would be really useful (like triggering an extract refresh). For now, you can experiment with the Vizportal API.

This guide is by no means complete and I’m sure as Tableau adds new functionality there will be updates to be made. Consider this v0.1 with more to come in the future.

The Tableau Undocumented API Made Easy with Python

2016-12-14T00:00:00-08:00

After returning from this year’s Tableau Conference and all the great sessions on Tableau’s APIs, I couldn’t wait to dive into the content the devs made available. I have been collecting ideas for automated Tableau Server tools for a while now and TC16 ignited the passion in me to put some of them into action.

I started by exploring the REST API with Python and quickly realized that there was a gap between what I wanted to build and the functionality that the API offered. While many Server actions can be scripted, simple things like modifying the schedule attached to an Extract refresh isn’t possible. Why Tableau? You’ll let me change the schedule for a Subscription, so why not an Extract?

The answer is probably that the devs are super busy and they just haven’t gotten around to it. Hopefully this and other Extract functionality (like triggering Extract refreshes) is just a release or two away.

But what should I do in the mean time? I can’t just give up. This is a core element of my app. After a quick post to the Global Tableau Developers Group, I found a tiny light at the end of the tunnel in the form of Tableau’s “Undocumented” API.

What is the “Undocumented API”?

You’re probably not familiar with the Undocumented API by name but if you’ve ever used Tableau Server, you’ve used it. Put simply, the Undocumented API (hereafter referred to by its real name: the Vizportal API) is the technology that enables Tableau Server’s Web Client. Any time you interact with Tableau Server, your browser is issuing calls via the Vizportal API to execute the commands you choose. Using tools like Fiddler (or by inspecting the page with Chrome or Safari) you can see the calls that your browser is making to the Vizportal API.

By inspecting the page, we can see there is a call to Vizportal to "getExtractTasks"

This is Amazing! How Come I Never Knew this Existed?

From this we can clearly see that there are API methods for lots of tasks that are not part of the REST API. By leaving the inspector window open and clicking around Server, you can see exactly what calls are used to control each function. Amazing! How come I never knew this existed?

This is why the Vizportal API is known as the Undocumented API. While it’s technically possible to make these same calls from your own code, it’s not officially supported by Tableau. Tableau could disable or change these methods at any time, without notice so if you’re going to use this consider yourself warned.

How To Connect to the Vizportal API using Python

Even though it’s not supported or officially documented, there is some material online that discusses how you can connect to Vizportal. The team at Tableau that runs their internal server has a post that describes one method but it stops just short of providing the code. I’ll present a method using Python (version 2.7) but this could be done with Javascript or any other language the supports HTTP requests and RSA encryption. To log into the Vizportal API we’ll need to perform the following steps:

Generate a Public Key that we can use to encrypt our password that will be used for the login
Encrypt the user’s password with RSA PKCS1 encryption
Login to Vizportal and retain the workgroup_session_id and XSRF-TOKEN so we can submit other calls to Vizportal

Before I go any further I should point out that this is not a back door into Tableau Server. We are still authenticating with a credentialed user and as such, the requests we make will honor the permissions of the user that we’re logging in with. If you’re unable to successfully execute this, check that you’re using a licensed user account with permission to execute the calls you’re making.

Generating the Public Key

The first step we need to do is to generate a public key and capture the modulus and exponent so we can encrypt our password:

import os
import sys
import requests
import json

tab_server_url = "http://your_tableau_server_url"

def _encode_for_display(text):
     return text.encode('ascii', errors="backslashreplace").decode('utf-8')

# Establish a session so we can retain the cookies
session = requests.Session()

def generatePublicKey(): 
      payload = "{\"method\":\"generatePublicKey\",\"params\":{}}"
      endpoint = "generatePublicKey"
      url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
      headers = {
      'content-type': "application/json;charset=UTF-8",
      'accept': "application/json, text/plain, */*",
      'cache-control': "no-cache"
      }
      response = session.post(url, data=payload, headers=headers)
      response_text = json.loads(_encode_for_display(response.text))
      response_values = {"keyId":response_text["result"]["keyId"], "n":response_text["result"]["key"]["n"],"e":response_text["result"]["key"]["e"]}
      return response_values

This function will return a dictionary with 3 key/value pairs that we can use for the next step.

The keyId
The modulus “n”
The exponent “e”

Put this into practice like this:

 # Generate a pubilc key that will be used to encrypt the user's password
 public_key = generatePublicKey()
 pk = public_key["keyId"]

Encrypting the Password

This was the hardest part of the entire process as I couldn’t find any good examples of how to do this with Python. After spending hours looking for a solution, I reached out to Zen Master Tamás Földi who graciously agreed to help me. Here’s the code he shared:

# You'll need to install the following modules
# I used PyCrypto which can be installed manually or using "pip install pycrypto"
import binascii
import Crypto
from Crypto.PublicKey import RSA
from Crypto.Cipher import PKCS1_v1_5
from base64 import b64decode

# Encrypt with RSA public key (it's important to use PKCS11)
def assymmetric_encrypt(val, public_key):
     modulusDecoded = long(public_key["n"], 16)
     exponentDecoded = long(public_key["e"], 16)
     keyPub = RSA.construct((modulusDecoded, exponentDecoded))
     # Generate a cypher using the PKCS1.5 standard
     cipher = PKCS1_v1_5.new(keyPub)
     return cipher.encrypt(val)

The objective here is to supply your password as the first argument and then input the key retrieved from the previous step as the second argument. The output will be an encrypted string that we will use to log in.

# Encrypt the password used to login
encryptedPassword = assymmetric_encrypt(your_password,public_key)

Now that we have the encrypted password, we’re ready to log in.

Logging into Vizportal

The login combines the encryptedPassword with the keyId and the username to issue a request to Tableau Server. If successful, we need to capture the cookie from the response header and extract the workgroup_session_id and XSRF-TOKEN to use in any subsequent calls.

# Make sure to change your username
tableau_username = "Your Username"

def vizportalLogin(encryptedPassword, keyId):
     encodedPassword = binascii.b2a_hex(encryptedPassword) 
     payload = "{\"method\":\"login\",\"params\":{\"username\":\"%s\", \"encryptedPassword\":\"%s\", \"keyId\":\"%s\"}}" % (tableau_username, encodedPassword,keyId)
     endpoint = "login"
     url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
     headers = {
     'content-type': "application/json;charset=UTF-8",
     'accept': "application/json, text/plain, */*",
     'cache-control': "no-cache"
     }
     response = session.post(url, data=payload, headers=headers)
     return response

# Capture the response
login_response = vizportalLogin(encryptedPassword, pk)
# Parse the cookie
sc = login_response.headers["Set-Cookie"]
set_cookie = dict(item.split("=") for item in sc.split(";"))
xsrf_token, workgroup_session_id = set_cookie[" HttpOnly, XSRF-TOKEN"], set_cookie["workgroup_session_id"]

If you’ve made it this far you should be able to log in into Vizportal. Now comes for the fun part!

Using the Vizportal API: Update Extract Schedules

As I mentioned before, it’s currently not possible to change the Extract schedule associated with an Extract refresh using the REST API but it can be done with Vizportal. Before you try this, take a look again at the call being made in the browser. It will give you the format you’ll need to construct the payload for the HTTP request. I had to experiment with the formatting of the payload string to get it to match but once I did it worked perfectly.

# The task_id is the value from the "tasks" table in the Workgroup database 
# that corresponds to the Extract associated with your Workgroup/Datasource
task_id = "55"
# The schedule_id is the value from the "_schedules" table in the Workgroup database 
# for the schedule you want to change the task to use
schedule_id = "1"

# Use the xsrf_token you got from the login response cookie
def updateExtractSchedule(task_id, schedule_id, xsrf_token):
     payload = "{\"method\":\"setExtractTasksSchedule\",\"params\":{\"ids\":[\"%s\"], \"scheduleId\":\"%s\"}}" % (task_id, schedule_id)
     endpoint = "setExtractTasksSchedule"
     url = tab_server_url + "/vizportal/api/web/v1/"+endpoint
     headers = {
     'content-type': "application/json;charset=UTF-8",
     'accept': "application/json, text/plain, */*",
     'cache-control': "no-cache",
     'X-XSRF-TOKEN':xsrf_token
      }
     response = session.post(url, data=payload, headers=headers)
     print response.status_code
     return response

Success!

So Why Do This?

The current version of the REST API lacks some basic functionality that would be really useful to have. If you have a Server that doesn’t have a dedicated Administrator it can be hard to maintain the content and schedules as it grows. Automating these tasks can give your Tableau Server extra life and improve the experience for your users. I fully expect that more functionality will be added with each release and it’s only a matter of time before the REST API becomes fully featured. In the mean time, experiment with Vizportal and see what you can come up with.

Big thanks to Chris Toomey and Tamás Földi for all of their help in figuring this out.

My First Tableau Public Viz!

2016-10-28T00:00:00-07:00

October is a great time of year. It’s full of age-old traditions like the changing of the color of leaves, the return of the Pumpkin Spice Latte (not a personal favorite but I bet a few of you know what I mean) and who can forget about Halloween. The one I look forward to every year is the Major League Baseball playoffs and it’s culmination in the World Series. Every season brings unforgettable memories and this year’s playoffs have been exciting as ever. Who knows, maybe this is the year the Cubs can finally win it all!

Unfortunately for me, my New York Yankees didn’t make the postseason and I’ll have to wait see if they can pull it off next year. So while I count down the days until Spring Training 2017, I thought I’d take a look back in history and reflect on their tradition of winning. If you’re a fan like me (or just love baseball and data), take a few minutes to soak in the nostalgia of 114 years of New York Yankees history.

Choosing a Tableau Server Architecture

2016-07-21T00:00:00-07:00

Depending on your organization’s needs, there are a variety of ways that you can configure Tableau Server. Choosing the right architecture for your server deployment is a crucial first step and can make all the difference in your users’ experience, and that of your administrator as well. So how do you know what configuration is right for you?

Single Node or Distributed Configuration

The decision to go with a Single Node installation or a Distributed (clustered) configuration should be based around a combination of the following details:

Total User Count (the maximum number of licensed users)
Concurrent Users (between 5 and 10% of your total user count should be a good enough estimate)
Your License Strategy
The Number of Extracts vs. Live Data Connections
The amount of load you plan to drive (What Drives Tableau Server Load?)
Availability expectations for your Tableau environment

So how many users do you have? Are you testing Tableau out with a select number of people in your organization or is your charter to grant access to everyone for ultimate transparency? In either case, there is a licensing strategy for all deployments and choosing the best one will also influence how you set up your configuration.

User-Based or Core Based License

User-based licensing is exactly what it sounds like. You purchase a license for every named user on your server. This is a common license strategy because most installations start out small and then grow organically. You may only have a small handful of users to begin with but this can quickly grow to 25, 50, 100+. For these environments, a single node running 8 cores with 32GB (the bare minimum) is generally sufficient and user-based licensing is perfectly suited.

But what happens when the entire company wants to get on the server? How can you add additional users when you have a limited amount of licenses? Wouldn’t it be great if you could automatically provision new users to a team that’s growing rapidly? That’s where core-based licensing comes in.

In contrast to user-based licensing, core-based licensing is based on the physical cores of your server’s hardware. You pay based on the processor cores of your hardware and the only limit to the number of users you can add is the level of performance you can consistenly deliver. With core-based licensing, you can onboard entire teams and quickly scale to meet your company’s demand. This license strategy assume a large volume of users and is most often associated with enterprise deployments.

Now that you’ve determined what licensing strategy is best for your deployment, it’s time to think about performance. As adoption increases, the volume of content on your server is going to grow and it’s quite common for there to be a perceived slow down in performance. Put simply, the demand for content (# of visualizations, extract refreshes, subscriptions being sent) has outgrown the original hardware specification and it’s time to upgrade.

So should you scale up or scale out?

Scaling Up

Scaling up means adding additional capacity within your current hardware. This could mean increasing the number of cores available to Tableau Server processes, adding more RAM to the system, swapping out hard disks for solid state drives or other improvements. Each of the above methods of scaling up comes with its own costs and some are greater than just the cost of the hardware. If you want to add additional cores to your server, you’ll need to upgrade your server license to leverage them and there’s a cost associated with that too. Scaling up can provide significant runway for your server and modern enterprise hardware can be upgraded several times. Eventually you’ll reach the limit of what a single machine can do. So what then? That’s where scaling out comes into play.

Scaling Out

Scaling out refers to adding additional capacity by increasing the total number of machines available to your Tableau Server configuration. Tableau Server supports a distributed or clustered configuration by letting you add additional machines to spread the load across them and thus increasing the overall capacity of your environment. As you can imagine, this can lead to some pretty complex configurations but for simplicity sake I’ll touch on just a couple of reasons why you might want to consider scaling out as an option:

Additional Capacity

The most common reason I see for scaling out is to add additional capacity to Tableau Server. If you’ve maxed out your single node and performance is still not where you want it to be, adding a 2nd or 3rd node can provide the additional horsepower you need. Tableau Server does this through the power of Worker nodes. Worker nodes act in concert with the Primary node and can share the load of your Tableau environment.

There are several clustered configurations that you can adopt and the one you choose mostly depends on how your users interact with content on the server. If your server has a lot of extract refreshes being performed daily, consider using a dedicated worker node for background processing.

Tableau’s Backgrounder process is a primary driver of load on Tableau Server (see Paul Banoub’s excellent post on All About the Backgrounder for more detail) and offloading it to a separate machine can yield benefits including:

Reduced strain on the VizQL process (which is responsible for rendering all those beautiful visualizations your team has built). Allocating more process threads to refreshing extracts, sending subscriptions, or executing commands via tabcmd. More process threads = more simultaneous jobs = less delay for content updates.

Example of a distributed system with a dedicated "Backgrounder" machine running 10 Background Processes

High Availability (HA)

High Availability is another major reason why I see organizations decide to scale out their servers. Tableau’s documentation does a great job of explaining what High Availability is so I won’t go into depth here but if you want to ensure that your Tableau Server is up even if one of your nodes goes down, you’re going to need a distributed architecture.

The best reason to scale out is if you want to build a high-performance, highly-available Tableau Server. These types of environments aren’t cheap but if you’re supporting thousands of users running thousands of extract refreshes around the clock it’ll give you the confidence to know that you can serve their needs.

Highly available configuration with external load balancer, 1 Primary, 2 Worker Nodes, 2 Background Nodes, 1 Backup Primary: 64 total cores

Bare Metal or VMs

Now that you’ve chosen your configuration, it’s time to decide what kind of hardware you’re going to deploy it on. Will you deploy on Bare Metal or use Virtual Machines (VMs)?

The decision on whether to use physical hardware or go with virtual machines is a major one that should be thought through carefully. Virtual machines can be a great way to go as they take less time to procure and set up and are often cheaper than ordering custom hardware. A word to the wise though…if you go with VMs for Tableau Server there are some strict criteria that you’ll need your system adminstrators to guarantee or managing your server’s reliability will be a nightmare:

Tableau’s minimum hardware requirements assume you are using bare metal physical servers, not virtual machines. The same is true for their scalability estimates.
Because Tableau Server is a resource-intensive and latency-sensitive application, it requires dedicated resources. Pooling or sharing of CPU and RAM across other VMs on a host machine will lead to lots of issues with your configuration. Avoid pooling and sharing at all costs.
Make sure you get 100% dedicated CPU allocation. Tableau’s minimum requirement for a Production environment is 8 physical cores. Tableau ignores hyperthreading so make sure when you ask for 8 cores you’re really getting the full amount.
100% dedication of RAM is important too. It cannot be dynamic RAM either. It should be contiguous on the host server and the minimum for a Production environment is 32GB. So if you’re running an HA setup with 3 machines you’ll need at least 96GB of dedicated RAM from a single VM host.
The high-latency of Tableau Server also requires fast disk access. Don’t settle for network attached storage (NAS). Insist that you’re given a tiered SAN to get the highest level of write speed.
If you’re running a distributed environment, you’ll want the latency between your workers to be less than 10ms. A poorly tuned VM setup will lead to poor server performance and you’ll feel like you wasted a lot of money on those extra cores you purchased.
Do not use VM snapshots for backing up Tableau Server. The only supported method for backing up Tableau is to use the ‘tabadmin backup’ command.
If your system admin is insisting on giving you a setup that has VMotion enabled, don’t use VMs. VMotion is a technology that allows VMs to move from one physical host to another for managing performance and it is not compatible with Tableau’s licensing technology. More about why this happens can be found here: VM server vs. Physical server.
Keep in mind that the above applies to all machines running Tableau Server. If you’re using a clustered setup, make sure you get the same guarantees for the Worker nodes and the Primary.

Putting it All Together

By now you should have a good idea of what type of license you’re going to use, what your configuration will look like and whether or not you’re going to use physical hardware or go with a slate of dedicated VMs. Every installation of Tableau Server is unique to the business cases and environment it serves. I’d love to hear about your experiences with single vs. clustered setups and if you’ve had challenges with VMs over bare metal (or vice versa) so please leave your stories in the comments.

What Drives Tableau Server Load?

2016-01-12T00:00:00-08:00

One of the most common challenges I see from clients using Tableau Server is around how they manage their server’s performance at scale. The story usually goes something like this:

Initial Install of Tableau Server (low number of users, low utilization, fast server performance)
Adoption grows (users increases moderately, utilization becomes more frequent, volume of content grows, fast performance maintained)
Adoption continues to increase (users added daily/weekly, daily content interaction/creation, regular update of content on server, noticeable but manageable performance declines witnessed)
Critical mass reached (100s to 1000s of users, constant content interaction, simultaneous updates of content occurring throughout the day, performance declines to a point where it impacts ability to conduct analysis)

Of course the desire is to never reach the point where the performance of your Tableau Server environment impacts the ability to do business, but it’s an unfortunate reality that you may have faced. So what do you do about it? To understand how to manage Tableau Server performance we first need to understand what drives it. For the remainder of this post we’ll refer to this as “load”.

What is “Load” and Why does it matter?

Load is defined as a measure of the amount of computational work that a computer system performs. It most commonly refers to CPU but can also refer to memory usage, network data transfer, and disk operations like I/O. We’ll focus on all of these as they are all impacted by Tableau Server’s application processes, albeit in varying degrees depending on the process.

What drives “Load” on Tableau Server

Under the hood of Tableau Server are several different application processes that combine to create the experience. You can learn more about them here but when we talk about what drives load we’re really talking about 3 processes:

Backgrounder - executes server tasks, including extract refreshes, “Run Now” tasks, subscriptions, and tasks from tabcmd
Data Engine (also known as the TDE Server) - Stores data extracts and answers queries
VizQL Server - loads and renders views, computes and executes queries

Each of these processes is responsible for a key part of the Tableau Server experience and they frequently work in combination to accomplish the demands of users consuming, updating, and publishing content.The degree to which each process impacts your server’s performance will depend on your Tableau Server architecture. For example, if your server has a large number of extracts being refreshed concurrently, the Backgrounder process is going to affect performance more heavily than a server that favors live data connections. We won’t dig into architecture decisions in this post but how you design your server (single vs. multi-node, dedicated node for Backgrounder processes vs. mixed use, high-availability, etc.) is also a key consideration for how your server performs. For now, let’s dig into the 3 processes above to understand their impact on load in greater detail.

Backgrounder

The Backgrounder process is responsible for a large number of tasks but the most commonly used ones are extract refreshes, “Run Now” tasks, subscriptions, and any command initiated from the tabcmd utility. Any time you refresh an extract on the Tableau Server (manually or via a schedule), it’s the Backgrounder process at work. By design, the Backgrounder process is a single-threaded process and there can be multiple Backgrounder processes running at the same time. If you’ve ever seen the Tableau Server Maintenance page this will look familiar:

In this configuration, there are 2 nodes in the server cluster and one of them is dedicated to Backgrounder processing with a total of 24 Backgrounder processes. This allows the server to execute 24 simultaneous Background Tasks.

It’s important to note that single-threaded does not mean that multiple processes can’t run in parallel. What it refers to is that a CPU core cannot be logically split to share the workload, a process known as hyper-threading. Tableau Server does not currently support hyper-threading because each Backgrounder process has the ability to consume 100% of the CPU resources for a given core. Instead, multiple Backgrounder processes are used and the server runs one task per process, on separate cores.

Having said all of that, the Backgrounder processes primarily drives CPU, I/O and network resources. A large extract refresh could consume a lot of the network resources due the transfer of large data volumes and running several of these simultaneously can cause CPU on the server to spike. We’ll discuss how you track and manage this in a bit.

Data Engine

The Data Engine is what powers the data in your visualizations. The VizQL server, API server, data server, and background server processes all rely on the Data Engine to load the data into memory where it can then be queried at high speed. This is what allows Tableau Server to perform some operations faster when using extracts as opposed to live connections. As you’ve probably guessed, the Data Engine primarily drives load in regard to Memory usage but under high load it can consume CPU and I/O and lead to performance bottlenecks. If taxed too much, a single instance of the Data Engine can consume all CPU resources while it is processing requests resulting in slow performance for users interacting with content. It is common during an extract refresh that the Data Engine will be employed to write the data to its repository.

VizQL Server

The VizQL Server is the process that loads and renders views & computes and executes queries. This is rendering engine that takes your data and turns it into the visualizations that you’ve designed. Every time you load a workbook or make a filter selection, the VizQL process is querying the Data Engine and rendering the results. Load can only be associated with this process from browser-based interactions meaning that it’s driven by users performing tasks on the Tableau Server. The more users you have interacting with the server at a given time (concurrency), the higher the load is going to be on the VizQL process(es). Most often when users complain that it takes a long time to load a workbook, it’s the VizQL process that is experiencing heavy load. VizQL consumes CPU, I/O, or network resources and these processes can sometimes run out of memory.

How Do I Measure “Load” and What can I do about it?

Now that we understand what drives load, we need to measure it to understand where the constraints are. Tableau has released an open-source tool call TabMon that captures performance statistics on your server’s hardware and presents it via a set of Tableau workbooks so you can analyze it and make changes to your server’s configuration. There’s also a great series from Paul Banoub on How to Monitor Your Tableau Server. Below is an example of a visualization I’ve used to track the CPU, Memory and Disk load on my server’s hardware:

CPU, Memory, and Disk utilization at 1 hour increments

Depending on what you learn from the results, you might just need to add a couple of additional processes but if you’re really at capacity an architecture change could also be in the cards. Understanding how Tableau Server performs at scale is critical to providing your users with the best experience possible. Whatever insights you uncover, it’s best to review them in the context of what your users are doing. Talk to them and find out where they are experiencing issues when using Tableau Server and see if the data corroborates their stories. Develop a plan to address those topics and remember to capture the evidence from your server so you can see if the changes made a difference.