You will have to run the following steps after destroying a service to clean up all the resources in DCOS. We are going to clean up the cassandra-px service in this example. These steps can be used to clean up any service in DCOS including the Portworx service.

Shutdown the service if it is still running

Run dcos service --inactive to find the ID of the service that you want to cleanup. The service should in inactive state, ie ACTIVE should be set to False.

$ dcos service --inactive
NAME                     HOST            ACTIVE  TASKS  CPU    MEM      DISK   ID
cassandra-px  ip-10-0-2-15.ec2.internal   False    2    6.7  27530.0  59890.0  cc3a8927-1aec-4a8a-90d6-a9c317f9e8c6-0051
marathon              True     4    3.1   3200.0    0.0    cc3a8927-1aec-4a8a-90d6-a9c317f9e8c6-0001
metronome             True     0    0.0    0.0      0.0    cc3a8927-1aec-4a8a-90d6-a9c317f9e8c6-0000
portworx      ip-10-0-2-15.ec2.internal   True     6    1.8   5632.0  16384.0  cc3a8927-1aec-4a8a-90d6-a9c317f9e8c6-0050

Then use the dcos service shutdown command to shutdown the service

$ dcos service shutdown cc3a8927-1aec-4a8a-90d6-a9c317f9e8c6-0051

Run the janitor script to clean up the reserved resources as well as any state stored in Zookeeper

$ SERVICE_NAME=cassandra-px
$ PRE_RESERVED_ROLE="your_pre_reserved_role/" # Set this if you started the service with a pre-reserved-role
$ dcos node ssh --master-proxy --leader "docker run mesosphere/janitor / -r ${PRE_RESERVED_ROLE}${SERVICE_NAME}-role -p ${SERVICE_NAME}-principal -z dcos-service-${SERVICE_NAME}"

Cleanup PX remnants from slave nodes

Stop the portworx service on all the nodes and remove the docker container

sudo systemctl stop portworx
sudo docker rm portworx.service -f

Remove the portworx service file from all the nodes

sudo rm /etc/systemd/system/portworx.service -f
sudo rm /etc/systemd/system/ -f
sudo rm /etc/systemd/system/ –f
sudo systemctl daemon-reload

Note: If you are going to re-install Portworx, you should wipe out the filesystem from the disks so that they can be picked up by Portworx in the next install. This can be done by running the following pxctl command

# Use with care since this will wipe data from all the disks given to Portworx
sudo /opt/pwx/bin/pxctl service node-wipe --all

If you are running Portworx version < 1.3, run the following command instead of node-wipe,

sudo wipefs -a /dev/sda123 # Replace with your disk names

Remove the Portworx config and files from all the nodes

sudo chattr -i /etc/pwx/.private.json
sudo rm -rf /etc/pwx
sudo umount /opt/pwx/oci
sudo rm -rf /opt/pwx

Also remove the Portworx kernel module from all the nodes

sudo rmmod px -f

If you have the dcos cli installed then you can execute the above steps on all the nodes by running the following script

ips=(`dcos node --json | jq -r '.[] | select(.type == "agent") | .id'`)
for ip in "${ips[@]}"
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo systemctl stop portworx'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo docker rm portworx.service -f'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rm /etc/systemd/system/portworx.service -f'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rm /etc/systemd/system/ -f'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rm /etc/systemd/system/ –f'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo systemctl daemon-reload'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo /opt/pwx/bin/pxctl service node-wipe --all'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo chattr -i /etc/pwx/.private.json'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rm -rf /etc/pwx'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo umount /opt/pwx/oci'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rm -rf /opt/pwx'
        dcos node ssh --mesos-id=${ip} --master-proxy 'sudo rmmod px -f'