SPEAK 3: Adventures of a backend developer

Those of us who tried creating Sitecore applications in Speak 1.1 and 1.2 and spend most of our days in Visual Studio writing C# code, attempting to create a Speak 3 application might be a bit of a challenge. It certainly was for me. Even though Speak 3 feels much easier and intuitive than its’ predecessors, it still has a learning curve for a backend developer who is not familiar with NodeJs and Angular.

Having implemented my first Speak 3 application, I felt that documenting what I had to go through might be beneficial not only for me 3 months down the road, but also for other developers. So here it is.

Prerequisites and tools

• Speak 3 is part of Sitecore 9 release, so to write applications in Speak 3 you need Sitecore 9.Sitecore 9 - https://dev.sitecore.net/Downloads/Sitecore_Experience_Platform/90/Sitecore_Experience_Platform_90_Initial_Release.aspx

The following items were used to create the Data Import application:

• NodeJs 8.9.1 (includes npm 5.5.1) - https://nodejs.org/en/
• Visual Studio Code - https://code.visualstudio.com/ with following plugins

• Sitecore provided a list of downloads at https://dev.sitecore.net/Downloads/Sitecore_SPEAK/3/Sitecore_SPEAK_3.aspx

And documentation for Speak 3 at https://doc.sitecore.net/SPEAK/90/SPEAK%203

There is a sample application that you can look at to see how an application should look like:

I have used packages.json from the sample application to install required node modules with versions that Speak 3 application expects. It is probably the easiest for a backend developer.

One thing in the development process that is very unusual for a backend developer and is worth mentioning. Speak 3 application development for the most part can happen outside of Sitecore in disconnected mode, which means there is no connection to Sitecore and you are working with frontend code exclusively. You can mock service responses though.

Creating Angular application

Sitecore Speak 3 uses Angular 4, which means that all modules you decide to use should support this version of Angular. In addition to that syntax is different between Angular versions, and you need to pay attention to that as well when writing your code.

You can follow the steps outlined in Quick Start guide on Angular site at https://angular.io/guide/quickstart to create an empty Angular 4 application.

Execution of these commands will give you a folder structure for an Angular application that you can start building your application upon.

I copied packages.json and .npmrc files from a sample Speak 3 application that Sitecore documentation provided into the root of my Angular application. If you are familiar with Angular, you might be able to install necessary modules one by one using Sitecore documentation, but it was a bit of a challenge for me, so I chose an easier route.

• In command prompt I executed
  cd C:\Speak\codefe 
  npm install
  That created node_modules folder in the root of my application and installed modules specified in packages.json file.
• At this point, I could open the application in Visual Studio Code. 

To define the port number my application should be running at, I added the following configuration entry in “defaults” property in angular-cli.json.

At first I created my Angular application without SASS. I was able to switch my project to use SASS by following instructions from http://brdjx.com/angular-4-with-sass/. If you switched to SASS and renamed existing .css files, make sure you updated references to scss files in @Component directive.

Running Angular application in disconnected mode

In addition to package versions, packages.json has a section that defines different commands to use for running applications in different modes.

I have been using mostly “start:disconnected” one during the development process. To run the application in this mode I have copied disconnected-mode.conf.js file into the root of my application folder:

To call this script and start the application in disconnected mode I executed the following command in command prompt:

npm run-script start:disconnected

After this command is executed, if there are no exceptions during compilation I was able to see my application at http://localhost:[port] where [port] is port number that is specified in angular-cli.json file in the root of application.

In my case I had the application running at http://localhost:4300

At that point, my application was ready for feature development!

Speak3 Application Page

When you open app.component.html file which is the starting point of the page structure for application, you would see the following:

 I replaced HTML in it like so (following sample application):

To create a new page component, I ran the following command in the command prompt where I executed previous commands:

ng generate component landing-page

Now I have a new folder with all the necessary files for my new landing-page component.

Before the page would load properly and look like a Sitecore application with Sitecore CSS, I had to change styles configurations in angular-cli.json file like so: 

I replaced styles.css with styles.scss and added ../node_modules/@speak/styling/dist/styles/sitecore.css

After that, I had to close the Visual Studio Code and reopen it again for the changes to take effect.

In the next step I added routing to the app.modules.ts file: 

This tells the application to load LandingPageComponent when the root of application is being loaded.

After these steps had been performed I could start working on the actual Landing Page component. First I made changes to HTML file like so: 

Then I had to add an implementation for properties and functions required by components I referenced in html: 

At this point my application was ready to be compiled and previewed. To do that I run the same command I ran before to start my angular application:

npm run-script start:disconnected

After the application is successfully compiled I was able to view it in the browser: 

To enable showing up of Logout section of the page, I copied three files into my mock folder from Sitecore sample application.

If you need to stop Angular application, in the command prompt window where you started it, press “Ctrl-C” and type Y. 

Building Components

Sitecore Angular Business Component Library package comes with a number of components that you can use. It took me some time to find the description for these components. You can find the list, description on how to use each component, and samples in \node_modules\@speak\ng-bcl\CHANGELOG.md inside your Angular application.

Any other components you need can be created the same way you would create any other Angular4 component. There is plenty of tutorials and blogs online for that.

You can use translation in your labels that will take values from Sitecore when the application runs in Sitecore context. 

For button labels the code looks like so:

While the application is still in disconnected mode, you would see something like this:

I was building a data import tool, and the page above shows the list of all available scripts for the import. My application calls Sitecore api to execute these scripts and save configurations that are created and edited in this application.

Creating of Sitecore Application items

To create a Speak3 Sitecore application I switched to core database and created a new Application item under /sitecore/client/Applications

Under the new node I created Translations and UserAccess items (the same as in sample application). In my Angular application in app.module.ts file I added/updated the following entries to match Sitecore IDs of my newly created items:

Packaging for deployment

Once I was happy with how application runs in disconnected mode it was time to bring it into Sitecore.

To do that I ran the following command in command prompt:

npm run-script build

This command created a dist folder in the root of my application. 

 

Deploying Application to Sitecore

After my dist folder was ready I copied its content into C:\inetpub\wwwroot\[Sitecore Site]\sitecore\shell\client\Applications\XcMigrationTool

The destination path should match the path to application in Sitecore tree and base-href value in the script parameters in package.json

At this point my Speak3 application is ready to be loaded in Sitecore context.

I hope this might turn out to be helpful for someone :)

Deleting Sitecore Items in SQL (PROCEED WITH CAUSION!)

Recently we had to delete a large number of items from Sitecore, and as we all know the task usually takes a long time to run when you kick it off from Content Editor.

If you absolutely must delete items quickly, you can try doing it in SQL. Bare in mind that it is not advisable to manipulate item data directly in the database, and if you are not very experienced in Sitecore, I would advise you not to use this script. However, if you know what you are doing, it might be handy.

So here you go:

DECLARE @parentId as uniqueidentifier;
DECLARE @RowsToProcess  int
DECLARE @CurrentRow     int
DECLARE @SelectCol1     uniqueidentifier
DECLARE @Items2Delete TABLE (RowID int not null primary key identity(1,1),ID uniqueidentifier)

set @parentId = '{38994C59-E601-4823-8377-3903940FCFCC}'

INSERT INTO @Items2Delete (ID)
SELECT [Descendant] as ID
  FROM [dbo].[Descendants]
  where Ancestor = @parentId
SET @RowsToProcess=@@ROWCOUNT
print  @RowsToProcess

INSERT INTO @Items2Delete (ID)
select ID from [dbo].[Items] where ParentID = @parentId
SET @RowsToProcess=@@ROWCOUNT
print  @RowsToProcess

SET @CurrentRow=0
WHILE @CurrentRow<@RowsToProcess
BEGIN
    SET @CurrentRow=@CurrentRow+1
    SELECT
        @SelectCol1=ID
        FROM @Items2Delete
        WHERE RowID=@CurrentRow

declare @id as uniqueidentifier;

set @id = @SelectCol1
print @id

exec sp_executesql N'DELETE FROM [Items]
  WHERE [ID] = @itemId

  DELETE FROM [SharedFields]
  WHERE [ItemId] = @itemId

  DELETE FROM [UnversionedFields]
  WHERE [ItemId] = @itemId

  DELETE FROM [VersionedFields]
  WHERE [ItemId] = @itemId',N'@itemId uniqueidentifier',@itemId=@id

exec sp_executesql N'DELETE FROM [Descendants] WHERE [Descendant] = @itemId',N'@itemId uniqueidentifier',@itemId=@id
exec sp_executesql N' DELETE FROM [Links] WHERE [SourceItemID] = @itemID AND [SourceDatabase] = @database',N'@itemID uniqueidentifier,@database nvarchar(6)',@itemID=@id,@database=N'master'
exec sp_executesql N'DELETE FROM [Tasks] WHERE [ItemID] = @itemID AND [Database] = @database',N'@itemID uniqueidentifier,@database nvarchar(6)',@itemID=@id,@database=N'master'

END

If you have done the same thing before and see any issue with this script, don't hesitate to comment. :)

Sitecore Data source creation automation

There are plenty of different ways architects and developers organize data sources for Sitecore page items. I prefer to reserve tree structure under Home item only for page item types, and have data source in a separate structure. If you don't have a predictable pattern to how your data source items are stored, it might be very difficult to find things. If your solution rely mostly on Experience Editor, it might not be such a big deal, but if you do use Content Editor a lot, having clear structure to your data sources, might save you time in the long run.

I have been using my own module for several projects now, that allows to control data source structure and their live cycle. You can download the module either from SitecoreMarketplace or from GitHub at https://github.com/jcore/JCore.Sitecore.DatasourceAutomation

Sitecore Datasource Automation Module creates datasource folders for corresponding page items outside of Website tree structure. When pages are created, renamed deleted or moved, the datasource folders follow the page items they are attached to.

Configuration

After the package is installed or unicorn sync from the solution you should see three new templates under Template/Foundation/Datasources:

In addition to templates you’ll find a new branch under Branches/Foundation/Datasources:

To allow an easy start, there is a sample Data Sources folder created under sitecore/Content. You don’t have to use this folder.

To switch the datasource folder the module uses, either change JCore.Foundation.Datasources.DefaultRenderingDatasourceLocation setting in Website/App_Config/Include/Project/JCore.Common.Website.config or set datasourceRootItem attribute of your site definition like so:

<site name="sample_datasources_site" patch:after="site[@name='modules_website']"

                  targetHostName="$(rootHostName)"

                  database="master"

                  virtualFolder="/"

                  physicalFolder="/"

                  rootPath="/sitecore/content/Sample"

                  startItem="/home"

...

                  datasourceRootItem="{3C1FB34D-6821-4D26-A4DF-1840AF5CFD2D}"/>

To allow for an item to have automated datasource, you would have to add _NonChildDatasourceSupport template to the list of base template for the item.

Now when you create an item based on updated template you should see the datasource folder created for your item, Datasource Folder field update to point to the datasource folder location and a new contextual menu at the tom with grayed out “Datasource” chunk.

If your page item has any presentation settings defined and any rendering there have Datasource Template set, the module will create items under datasource folder for that rendering and update datasource location in page item presentation settings to point to the newly created datasource item.

If you have existing items that you would like to have the same datasource automation functionality enabled, you can still add template inheritance. However, Datasource Folder field will be empty for all items that inherit this template. If you have datasource folders for these item, simply update the Datasource Folder field to point to these folders, otherwise, click “Create Data Source” at the top in Data Source chunk, and the datasource folder will be created for you.

The structure of datasource folders always follows the same pattern and follows the tree structure of page items.

If you choose to delete a page item, corresponding datasource folder will be deleted as well, but only if no other pages use its datasource items.If you move or rename page item, datasource folder will be renamed as well.

"Top Searches" functionality using Sitecore ReportingService

Sitecore "Internal Search" report provides information about top keywords that were used in your website searches. Why not use this report to show top searched phrases on the website? (Thank you Tony Wang for the idea)

First of all you have to track the search queries in DMS. There are plenty of blog posts describing how to do that, so I am not going to explain it. I'll simply provide the code that I used to do that:

        public virtual void TrackSiteSearch(Item pageEventItem, string query)
        {
            Assert.ArgumentNotNull(pageEventItem, nameof(pageEventItem));
            Assert.IsNotNull(pageEventItem, $"Cannot find page event: {pageEventItem}");
            if (this.IsActive)
            {
                var pageEventData = new PageEventData("Search", Constants.PageEvents.Search)
                {
                    ItemId = pageEventItem.ID.ToGuid(),
                    Data = query,
                    DataKey = query,
                    Text = query
                };
                var interaction = Tracker.Current.Session.Interaction;
                if (interaction != null)
                {
                    interaction.CurrentPage.Register(pageEventData);
                }
            }
        }

Where Costants.PageEvents.Search is defined in

    public struct Constants
    {
        public struct PageEvents
        {
            public static Guid Search => Guid.Parse("{0C179613-2073-41AB-992E-027D03D523BF}");
        }
    }

After we put the code in place to track the search queries, we can start implementing the "Top Searches" functionality. Before you do that, make sure that the tracked data is showing up in "Internal Search" report in  Experience Analytics.

Now to the report call implementation.

I have implemented two methods to retrieve results from the report. One calls the ReportingService passing ReportQuery object and encoding the response data to get the same results Experience Analytics report shows. The second one creates ReportQuery with parameters that are required to get the right results.

public IEnumerable GetTopSearchQueries()
{
 try
 {
  var reportingService = ApiContainer.Repositories.GetReportingService();
  var reportQuery = GetReportQuery();
  ReportResponse reportResponse = reportingService.RunQuery(reportQuery);
  var encoder = ApiContainer.GetReportResponseEncoder();
  var result = encoder.Encode(reportResponse);
  if (result != null && result.Data != null && result.Data.Localization != null 

&& result.Data.Localization.Fields != null && result.Data.Localization.Fields.Any())
  {
   var searchFields = result.Data.Localization.Fields.FirstOrDefault();
   if (searchFields != null)
   {
    return result.Data.Localization.Fields.FirstOrDefault().Translations.Select(r => r.Value).ToList();
   }
  }
 }
 catch (Exception ex)
 {
  Log.Error(ex.Message, ex, this);
  var obj = CreateObject("reporting/dataProvider");
 }
 return new List();
}

private ReportQuery GetReportQuery()
{
 ReportQuery reportQuery = new ReportQuery();
 reportQuery.Site = "SUM";
 reportQuery.Segments = new string[] { 

ID.Parse(Settings.GetSetting("Tracking.AllVisitsByLocalSearchKeyword", 

"{8A86098B-2A7A-42CD-8B62-EC5AF1BE4D42}")).ToShortID().ToString() };
 reportQuery.Keys = new string[] { "ALL" };
 reportQuery.Fields = null;
 reportQuery.Parameters.DateFrom = DateTime.Now.AddMonths(-6);
 reportQuery.Parameters.DateTo = DateTime.Now.AddDays(1);
 reportQuery.Parameters.TimeResolution = TimeResolution.Collapsed;
 reportQuery.Parameters.KeyTop = 8;
 reportQuery.Parameters.KeySkip = 0;
 reportQuery.Parameters.PadEmptyDates = true;
 reportQuery.Parameters.KeyOrderBy = new FieldSort() { Direction = SortDirection.Desc, Field = SortField.Count };
 reportQuery.Parameters.KeyFromParent = string.Empty;
 reportQuery.Parameters.KeyFromAncestor = string.Empty;

 // RequestType is internal property, but it is required, so settings it through reflection
 var propertyInfo = reportQuery.GetType().GetProperty("RequestType", 

BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
 propertyInfo.SetValue(reportQuery, 

System.Convert.ChangeType(Enum.Parse(propertyInfo.PropertyType,"0"), 

propertyInfo.PropertyType), null);

 return reportQuery;
}

Debugging Sitecore assemblies using dotPeek

Recently I had to figure out the right parameter set that a Sitecore reporting Api required to produce results I needed, and the only way to determine that was to step through Sitecore assemblies.

Google search produces a very helpful post by Igal Tabachnik at
https://hmemcpy.com/2014/07/how-to-debug-anything-with-visual-studio-and-jetbrains-dotpeek-v1-2/

Following the steps and this post, plus selecting "Enable .NET Framework source stepping" in General tab under Debug (see blow) made it possible.

Attach to Process as you normally do and land on specified break point. 

Posting Sitecore MVC forms with Html.FormHandler while maintaining rendering context.

Recently I have run into a situation where I had to post back a form that was part of controller rendering. I was using Html.FormHandler with controller and action names specified. If the form doesn't have any validation errors, it works just fine, but if the server side validation fails on the post back, you will see several issues:

• If you return View in the controller rendering, you will see just the view returned without layout. Martina Welander described this scenario in her blog post at https://mhwelander.net/2014/05/30/posting-forms-in-sitecore-mvc-part-2-controller-renderings/
• If you have any placeholders in the view for your controller rendering, the placeholder will throw an exception.

There are a couple of solutions that have been described in various resources, but none of them solved the issue that I was facing to my liking, so I came up with my own one. It might be more radical than others, but worked perfectly for the issues that I was trying to solve.

By default if you use Html.FormHandler with controller and action names specified, Sitecore executes ExecuteFormHandler pipeline processor where ControllerRunner is being executed. The result is being saved into the output stream and "scOutputGenerated" route value is being set to 1. Setting these values excludes rendering from being executed executed and rendering context for it from being set. As a result placeholder that the view controller returns, might throw an exception.

Here is the overview of solution that I had arrived at.

1. By default sitecore FormHandler helper method renders scController and scAction hidden fields. I added another hidden field to the form called scRenderingUniqueId where I render rendering ID.
   
   

   @Html.CustomSitecore().FormHandlerRendering(Model.Rendering)

   
   Custom Helper:
   

       public class CustomSitecoreHelper : Sitecore.Mvc.Helpers.SitecoreHelper
       {
           protected HtmlHelper HtmlHelper { get; set; }
           public CustomSitecoreHelper(HtmlHelper htmlHelper)
               : base(htmlHelper)
           {
               Assert.ArgumentNotNull((object)htmlHelper, "htmlHelper"); 
               this.HtmlHelper = htmlHelper;
           }
           public virtual HtmlString FormHandlerRendering(Rendering rendering)
           {
               if (rendering == null)
                   return new HtmlString(string.Empty);
               var str = this.HtmlHelper.Hidden("scRenderingUniqueId", rendering.UniqueId).ToString();
               return new HtmlString(str);
           }
       }
   

   
   
   
2. I replaced ExecuteFormHandler pipeline processor with a custom version. That allowed my form post rendering to be processed in the same manner as all other renderings if scRenderingUniqueId is specified. As a result renderings with rendering ID specified in a hidden field rendering context was created.
   
   

   <mvc.requestBegin>
       <processor type="Neb.SitecoreExtensions.Pipelines.Request.RequestBegin.CustomExecuteFormHandler, Neb.SitecoreExtensions"  patch:instead="processor[@type='Sitecore.Mvc.Pipelines.Request.RequestBegin.ExecuteFormHandler, Sitecore.Mvc']"/>
   </mvc.requestBegin>

   
   

   public class CustomExecuteFormHandler : RequestBeginProcessor
       {
           public override void Process(RequestBeginArgs args)
           {
               HttpContextBase httpContext = args.PageContext.RequestContext.HttpContext;
               if (WebHelper.GetRequestType(httpContext) != HttpVerbs.Post)
                   return;
               this.ExecuteHandler(httpContext.Request.Form, args);
           }
   
           protected virtual void ExecuteHandler(NameValueCollection formValues, RequestBeginArgs args)
           {
               if (!string.IsNullOrEmpty(formValues["scRenderingUniqueId"]))
                   return;
               Tuple<string, string> controllerAndAction = MvcSettings.ControllerLocator.GetControllerAndAction(formValues["scController"].OrIfEmpty(MvcSettings.DefaultFormControllerName), formValues["scAction"]);
               if (controllerAndAction == null)
                   return;
               ExecuteHandler(controllerAndAction.Item1, controllerAndAction.Item2, args);
           }
   
           protected virtual void ExecuteHandler(string controllerName, string actionName, RequestBeginArgs args)
           {
               string str = new ControllerRunner(controllerName, actionName).Execute();
               if (str.IsEmptyOrNull())
                   return;
               RequestContext requestContext = PageContext.Current.RequestContext;
               requestContext.HttpContext.Response.Output.Write(str);
               requestContext.RouteData.Values["scOutputGenerated"] = (object)"1";
           }
       }
   

   
   
   
3. I added a new GetPostControllerRenderer pipeline processor to mvc.getRenderer pipeline after GetControllerRenderer. 
   
   

   <mvc.getRenderer>
           <processor patch:after="*[@type='Sitecore.Mvc.Pipelines.Response.GetRenderer.GetControllerRenderer, Sitecore.Mvc']" type="JCore.SitecoreExtensions.Pipelines.Response.GetRenderer.GetPostControllerRenderer, JCore.SitecoreExtensions" />
   </mvc.getRenderer>

   
   
   GetPostControllerRenderer pipeline processor:
   
   

   public class GetPostControllerRenderer : GetRendererProcessor
   {
           public override void Process(GetRendererArgs args)
           {
               HttpContextBase httpContext = args.PageContext.RequestContext.HttpContext;
               if (WebHelper.GetRequestType(httpContext) != HttpVerbs.Post)
                   return;
               var result = this.GetRenderer(args.Rendering, args, httpContext.Request.Form);
               if (result != null)
                   args.Result = result;
           }
   
           protected virtual Tuple<string, string> GetControllerAndAction(Rendering rendering, GetRendererArgs args, NameValueCollection formValues)
           {
               Tuple<string, string> controllerAndAction = MvcSettings.ControllerLocator.GetControllerAndAction(formValues["scController"].OrIfEmpty(MvcSettings.DefaultFormControllerName), formValues["scAction"]);
               if (controllerAndAction == null)
                   return (Tuple<string, string>)null;
               return MvcSettings.ControllerLocator.GetControllerAndAction(controllerAndAction.Item1, controllerAndAction.Item2);
           }
   
           protected virtual Renderer GetRenderer(Rendering rendering, GetRendererArgs args, NameValueCollection formValues)
           {
               Tuple<string, string> controllerAndAction = this.GetControllerAndAction(rendering, args, formValues);
               if (controllerAndAction == null)
                   return (Renderer)null;
               string str1 = controllerAndAction.Item1;
               string str2 = controllerAndAction.Item2;
               if (args.Result is ControllerRenderer && rendering != null)
               {
                   if (rendering.UniqueId.ToString() != formValues["scRenderingUniqueId"])
                       return (Renderer)null;
   
                   return (Renderer)new ControllerRenderer()
                   {
                       ControllerName = str1,
                       ActionName = str2
                   };
               }
               return (Renderer)null;
           }
   }
   

   
   
4. To eliminate errors due to rendering context not being set, I also replaced RenderAddedContent with a custom version where I check if args.OwnerRendering and args.OwnerRendering.Properties are null to prevent exception that out-of-the-box version throws when placeholder is being rendered without rendering context.

After above changes were implemented the code stopped throwing exceptions due to rendering context being null and requests are being submitted to the proper controller action.