Continuous Integration & Deployment With Sitecore

Update 2: A full update and review of the original post below.
Update: Changes to how the baseline Git tag is moved in TeamCity due to the VCS labelling options only supporting moving of tags with CVS.

Not too long ago at Lightmaker we started to implement continuous integration & deployment/delivery. Before we set about deciding on which software or applications we would use we set out some goals.

  • Create an automated pipeline from the point of committing committing changes into version control to having them deployed on a target environment.
  • Use as few applications or pieces as software as possible.
  • Use as little scripting as possible.

Our goals are quite broad and simple but at Lightmaker we do both front-end and technical work so it was important that our front-end developers could use and understand as much of the process as possible.

When it comes to site files this is not such a difficult task as generally speaking you just deploy your packaged up files and you’re done! Of course this is an extreme simplification as in the real world you should deal with making sure files that are no longer needed are cleaned up, backing up before deploying etc.  So we found that deploying the files was quite straight forward, however it got a little more tricky when it came to deploying changes we made in our development Sitecore instances to any of our environments which we wanted to automate deployments. So we went about finding a solution that works for us, and this is key in that whatever solution you put in place should work for you! At this point we had already settled on what our “spine” would be (Git, TeamCityOctpus Deploy & Team Development for Sitecore), so we needed to find a solution which fit with our spine. I’m going to outline what where we ended up and some steps that should get you up and running. The steps focus on the Sitecore element rather than deploying website files and assumes you have some knowledge of Git, TeamCity, Octopus Deploy & Team Development for Sitecore already.

The following outlines all the tools, software and applications  that we eventually settled upon.

  • Git – version control.
  • TeamCity – build server.
  • Octopus Deploy – deployment server.
  • Team Development for Sitecore – uses Sitecore’s serialisation to create files on disk that can then be put into version control. Has an excellent plugin for Visual Studio for synchronising what is in version control with a Sitecore instance.
  • Sitecore Courier – used to Sitecore update packages based on serialised files from Sitecore.

So without further a-do the steps that should get you going!

Version Control – Git

Create a new tag on the branch you want to be deployed which will represent the commit that has last been deployed. For example we’re going to use the “develop” branch (because we use Git Flow) and so we’ve created a “develop-baseline” tag. If you’re doing this on a repository which has already been deployed then you should put the tag on the last commit which was deployed. Otherwise it will be the latest commit. The baseline commit is used to work out what changes have been made to the serialised files and the commit which is going to be deployed.

Build – TeamCity/Octopus Deploy

We’ll assume you have a project and build configuration created already! One thing to note, in steps below I haven’t included an MSBuild step at the start and you will more than likely have on! If you do you’ll need to have a copy of TDS installed on the save server as TeamCity.

Setting up the version control settings

  1. Create a VCS root for your branch that is to be deployed
  2. Create a VCS root for the baseline tag you created. Instead of checking out a branch you’ll need to use the tag that has been created. To do this set the following fields as follows;
    1. Default branch: “refs/tags/<baseline-tag-name>
    2. Use tags as branches: Checked
  3. We only need the serialised files from the baseline tag checkout and so we need to setup a couple checkout rules. Edit the checkout rules and add the following two lines;+:<TDS-serialised-files-folder>=>./<folder-to-put-serialised-files>

    1. <TDS-serialised-files-folder> – replace this with the location of your serialised files from TDS
    2. <folder-to-put-serialised-files> – replace this with a location of your choice, it’s where the serialised files from the baseline commit will be put.
    3. The second line will exclude everything except what we’ve specified prior.
  4. Change the VCS labelling mode to “Successful only” and add in the name of the tag used for the baseline commit and tick the checkbox for the branch that is to be deployed. TeamCity will move the tag to the commit which is to be deployed if everything has been successful for the build.

Setting up the build steps

  1. Create Sitecore Update Package – this step will call Sitecore Courier, you’ll need to build source files to get the executable and then put it in a location that TeamCity can access. Sitecore Courier will do a diff between the source and target serialised files and then output a Sitecore update package
    1. Runner type: Command Line
    2. Run: Custom script
    3. Custom script:mkdir %system.teamcity.build.workingDir%\courier-output\
      <folder-of-Sitecore-Courier-executable>\Sitecore.Courier.Runner.exe /source:%system.teamcity.build.workingDir%\baseline-tds /target:%system.teamcity.build.workingDir%\<TDS-serialised-files-folder> /output:%system.teamcity.build.workingDir%\courier-output\<Sitecore-update-package-name>.update
  2. Create Sitecore Deployment Package – this step will create a NuGet package that contains the .update file created in the “Create Sitecore Update Package” step. You’ll need to have a .nuspec file so that TeamCity can create the NuGet package, you can download this example.nuspec file if you haven’t created one before.
    1. Runner type: NuGet Pack
    2. Specification files: <nuspec-file-location>
    3. Output directory: \
      1. Publish created packages to build artifacts – Checked
  3. Deploy Packages – this step will call Octopus Deploy to trigger a release, you’ll need a project and at least one environment set up for this step. If you haven’t used TeamCity together with Octopus Deploy then I suggest checking out Octopus Deploy’s integration guide. Don’t worry about the OctoPack, Octopus will happily deploy bog standard NuGet packages!
    1. Runner type: OctopusDeploy: Release

Deploy – Octopus Deploy

A few more assumptions here! You need to make sure Octopus Deploy can read the NuGet package feed from TeamCity otherwise you won’t be able to select any of the NuGet packages created by TeamCity. Conversely you need to make sure TeamCity is set up to server up its NuGet feed :)!

  1. Deploy Sitecore Update Package – this step will simply retrieve the NuGet package created by TeamCity and extract it at the target environment. The NuGet package name should be whatever you’ve specified in the <id> node of the .nuspec file.
    1. Step type: Deploy a NuGet package
  2. Install Sitecore Update Package
    1. Put the TDS Package Installer somewhere on the server that the Octopus web application is installed on. If you’re not familiar with this little gem of a tool it’s actually a Visual Studio solution which Hedgehog provide along with TDS. You’ll need to compile the solution and grab the executable if you haven’t done this before! For more help on this there’s a question on the FAQ page for TDS, just find the question “How do I install an update package”.
    2. Set up an environment in Octopus which points to the server that the Octopus web application is installed on. This does mean you’ll need to install an Octopus Tentacle on the same server as the Octopus web application but this is ok, there’s no other way!
    3. Create a step to run a PowerShell script and have it call the TDS Package Installer executable

                                                   i.      Step type: Run a PowerShell script

                                                 ii.      Execute on roles: <environment-name-setup-previously>

                                                iii.      PowerShell script:

&”<TDS-PackageInstaller-location>” -v -sitecoreUrl “<target-Sitecore-instance-URL>” -sitecoreDeployFolder “<an-empty-folder>” -packagePath “<Sitecore-update-package-location>

 <TDS-PackageInstaller-location> – the location of the PackageInstaller.exe file, this should be on the same server as the Octopus web application (not the server you’re deploying to).

 <an-empty-folder> – I’ve left this in here because by default the solution provided by Hedgehog tries to install its connector to the directory of the website you’re deploying to. For us however we didn’t need to do this so we’ve removed this step from the code. Unless you intend on having the PackageInstaller executable on each target environment there’s no reason to use this parameter! What you will need to do is make sure the connector files it tries to install are present in your website directory. The two files below are what it tries to copy and where, both of which can be found in the output after building.

  1. HedgehogDevelopment.TDS.PackageInstallerService.dll to the bin folder of your website
  2. TdsPackageInstaller.asmx to the _DEV folder of your website, this probably won’t exist so you’ll need to create the folder.

So that should conclude the rough steps you need to do to get the pipeline in place, the only thing left to do is…

Test it out!

Grab newly created items in source Sitecore instance

Grab items in source Sitecore instance and save them into the TDS project. A good tip here is to use Sitecore Rocks! as TDS can hook into it and it will mimic the actions you do in Sitecore Rocks! in the TDS project e.g. if you add a template it will also add it. This way you can avoid having to do the sync like above!

Commit your changes into Git, do a push and then hope it all works :)!

Hopefully this gets your Sitecore CI/CD jump started, for us it’s working great! There are plenty of refinements that can be done and I suspect it will need to be tailored to suit your needs :). I’ll be posting some supplementary short(er) blogs regarding some bolt-ons and refinements such as using environment variables that can be passed through instead of hard coding paths in the places where scripts have been used.

Jason Bert